Skip to main content
SpringerLink
Log in

Continuous localization and mapping of a pan–tilt–zoom camera for wide area tracking

  • Original Paper
  • Published:
Machine Vision and Applications Aims and scope Submit manuscript

Abstract

Pan–tilt–zoom (PTZ) cameras are well suited for object identification and recognition in far-field scenes. However, the effective use of PTZ cameras is complicated by the fact that a continuous online camera calibration is needed and the absolute pan, tilt and zoom values provided by the camera actuators cannot be used because they are not synchronized with the video stream. So, accurate calibration must be directly extracted from the visual content of the frames. Moreover, the large and abrupt scale changes, the scene background changes due to the camera operation and the need of camera motion compensation make target tracking with these cameras extremely challenging. In this paper, we present a solution that provides continuous online calibration of PTZ cameras which is robust to rapid camera motion, changes of the environment due to varying illumination or moving objects. The approach also scales beyond thousands of scene landmarks extracted with the SURF keypoint detector. The method directly derives the relationship between the position of a target in the ground plane and the corresponding scale and position in the image and allows real-time tracking of multiple targets with high and stable degree of accuracy even at far distances and any zoom level.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Notes

  1. In the case of a PTZ sensor, the homography between each keyframe and the reference keyframe is the infinite homography \(\mathtt {H}_\infty \) that puts in relation vanishing lines and vanishing points between the images.

References

  1. Agapito, L., Hayman, E., Reid, I.D.: Self-calibration of rotating and zooming cameras. Int. J. Comput. Vision 45(2), 107–127 (2001)

    Article  MATH  Google Scholar 

  2. Arth, C., Klopschitz, M., Reitmayr, G., Schmalstieg, D.: Real-time self-localization from panoramic images on mobile devices. In: Proceedings of IEEE International Symposium on Mixed and Augmented Reality (2011)

  3. Barceló, L., Binefa, X., Kender, J.R.: Robust methods and representations for soccer player tracking and collision resolution. In: Proceedings of the International Conference on Image and Video Retrieval (2005)

  4. Bay, H., Tuytelaars, T., Gool, L.V.: SURF: speeded up robust features. In: Proceedings of the European Conference on Computer Vision (2006)

  5. Bernardin, K., Stiefelhagen, R.: Evaluating multiple object tracking performance: the CLEAR MOT metrics. J. Image Video Process. 2008, 1–10 (2008)

  6. Breitenstein, M., Reichlin, F., Leibe, B., Koller-Meier, E., Van Gool, L.: Online multi-person tracking-by-detection from a single, uncalibrated camera. IEEE Trans. Pattern Anal. Mach. Intell. 33(9), 1820–1833 (2011)

    Article  Google Scholar 

  7. Brendel, W., Amer, M., Todorovic, S.: Multiobject tracking as maximum weight independent set. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (2011)

  8. Civera, J., Davison, A.J., Magallon, J.A., Montiel, J.M.M.: Drift-free real-time sequential mosaicing. Int. J. Comput. Vision 81(2), 128–137 (2009)

    Article  Google Scholar 

  9. Collins, R.T., Tsin, Y.: Calibration of an outdoor active camera system. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (1999)

  10. Criminisi, A., Reid, I., Zisserman, A.: A plane measuring device. Image Vis. Comput. 17(8), 625–634 (1999)

    Article  Google Scholar 

  11. Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2005)

  12. Davis, J., Chen, X.: Calibrating pan-tilt cameras in wide-area surveillance networks. In: Proceedings of IEEE International Conference on Computer Vision (2003)

  13. Del Bimbo, A., Lisanti, G., Masi, I., Pernici, F.: Device-tagged feature-based localization and mapping of wide areas with a ptz camera. In: Proceedings of CVPR Workshops, Socially Intelligent Surveillance and Monitoring (2010)

  14. Del Bimbo, A., Lisanti, G., Masi, I., Pernici, F.: Continuous recovery for real time pan tilt zoom localization and mapping. In: Advanced Video and Signal-Based Surveillance (AVSS) (2011)

  15. Del Bimbo, A., Lisanti, G., Pernici, F.: Scale invariant 3D multi-person tracking using a base set of bundle adjusted visual landmarks. In: Proceedings of ICCV Workshops, International Workshop on Visual Surveillance (2009)

  16. Fischler, M.A., Bolles, R.C.: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 24(6), 381–395 (1981)

    Article  MathSciNet  Google Scholar 

  17. Fitzgerald, R.J.: Track biases and coalescence with probabilistic data association. IEEE Trans. Aeros. Electron. Syst. AES-21(6), 822–825 (1985). doi:10.1109/TAES.1985.310670

  18. Hartley, R.: Self-calibration from multiple views with a rotating camera. In: Proceedings of the European Conference on Computer Vision (1994)

  19. Hayman, E., Thorhallsson, T., Murray, D.W.: Zoom-invariant tracking using points and lines in affine views—an application of the affine multifocal tensors. In: Proceedings of the International Conference on Computer Vision (1999)

  20. Kang, S., Paik, J.K., Koschan, A., Abidi, B.R., Abidi, M.A.: Real-time video tracking using PTZ cameras. In: Proceedings of International Conference on Quality Control by Artificial Vision (2003)

  21. Klein, G., Murray, D.: Parallel tracking and mapping for small AR workspaces. In: Proceedings of the IEEE and ACM International Symposium on Mixed and Augmented Reality (2007)

  22. Leibe, B., Leonardis, A., Schiele, B.: Robust object detection with interleaved categorization and segmentation. Int. J. Comput. Vision 77(1), 259–289 (2007)

    Google Scholar 

  23. Liebowitz, D., Zisserman, A.: Metric rectification for perspective images of planes. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (1998)

  24. Lim, H., Sinha, S., Cohen, M., Uyttendaele, M., Kim, H.J.: Real-time monocular image-based 6-dof localization. Int. J. Robot. Res.: IJRR 34(4–5), 476–4925 (2015)

    Article  Google Scholar 

  25. Lovegrove, S., Davison, A.J.: Real-time spherical mosaicing using whole image alignment. In: Proceedings of European Conference on Computer Vision (2010)

  26. Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60(2), 91–110 (2004)

    Article  Google Scholar 

  27. Okuma, K., Taleghani, A., de Freitas, N., Little, J.J., Lowe, D.G.: A boosted particle filter: multitarget detection and tracking. In: Proceedings of the European Conference on Computer Vision (2004)

  28. Pernici, F., Del Bimbo, A.: Object tracking by oversampling local features. IEEE Trans. Pattern Anal. Mach. Intell. 36(12), 2538–2551 (2014)

    Article  Google Scholar 

  29. Seo, Y., Choi, S., Kim, H., Hong, K.S.: Where are the ball and players? Soccer game analysis with color based tracking and image mosaick. In: Proceedings of the International Conference on Image Analysis and Processing (1997)

  30. Sinha, S., Pollefeys, M.: Towards calibrating a pan-tilt-zoom cameras network. In: Proceedings of ECCV Workshops, Omnidirectional Vision and Camera Networks (2004)

  31. Sinha, S., Pollefeys, M.: Pan-tilt-zoom camera calibration and high-resolution mosaic generation. Comput. Vis. Image Underst. 103(3), 170–183 (2006)

    Article  Google Scholar 

  32. Song, D., Goldberg, K.: A minimum variance calibration algorithm for pan-tilt robotic cameras in natural environments. In: IEEE International Conference on Robotics and Automation (2006)

  33. Tao, H., Sawhney, H.S., Kumar, R.: Object tracking with bayesian estimation of dynamic layer representations. IEEE Trans. Pattern Anal. Mach. Intell. 24(1), 75–89 (2002)

    Article  Google Scholar 

  34. Tordoff, B., Murray, D.: Reactive control of zoom while fixating using perspective and affine cameras. IEEE Trans. Pattern Anal. Mach. Intell. 26(1), 98–112 (2004)

    Article  Google Scholar 

  35. Varcheie, P., Bilodeau, G.A.: Active people tracking by a PTZ camera in ip surveillance system. In: IEEE International Workshop on Robotic and Sensors Environments (2009)

  36. Williams, B., Klein, G., Reid, I.: Real-time SLAM relocalisation. In: Proceedings of the IEEE International Conference on Computer Vision (2007)

  37. Wu, Z., Radke, R.: Keeping a pan-tilt-zoom camera calibrated. IEEE Trans. Pattern Anal. Mach. Intell. 35(8), 1994–2007 (2013)

  38. Yang, B., Huang, C., Nevatia, R.: Learning affinities and dependencies for multi-target tracking using a CRF model. In: Proceedings of IEEE Conference on Computer Vision and Patter Recognition (2011)

  39. Cai, Y.,de Freitas, N., Little, J.: Robust visual tracking for multiple targets. In: Proceedings of the European Conference on Computer Vision (2006)

Download references

Acknowledgments

This work is partially supported by THALES Italia Spa, Florence, Italy.

Author information

Authors and Affiliations

  1. Media Integration and Communication Center, University of Florence, Viale Morgagni 65, 50134, Florence, Italy

    Giuseppe Lisanti, Iacopo Masi, Federico Pernici & Alberto Del Bimbo

Authors
  1. Giuseppe Lisanti
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Iacopo Masi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Federico Pernici
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Alberto Del Bimbo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Giuseppe Lisanti.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lisanti, G., Masi, I., Pernici, F. et al. Continuous localization and mapping of a pan–tilt–zoom camera for wide area tracking. Machine Vision and Applications 27, 1071–1085 (2016). https://doi.org/10.1007/s00138-016-0799-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00138-016-0799-x

Keywords

Search

Navigation