Abstract
Pan–tilt–zoom (PTZ) cameras are well suited for object identification and recognition in far-field scenes. However, the effective use of PTZ cameras is complicated by the fact that a continuous online camera calibration is needed and the absolute pan, tilt and zoom values provided by the camera actuators cannot be used because they are not synchronized with the video stream. So, accurate calibration must be directly extracted from the visual content of the frames. Moreover, the large and abrupt scale changes, the scene background changes due to the camera operation and the need of camera motion compensation make target tracking with these cameras extremely challenging. In this paper, we present a solution that provides continuous online calibration of PTZ cameras which is robust to rapid camera motion, changes of the environment due to varying illumination or moving objects. The approach also scales beyond thousands of scene landmarks extracted with the SURF keypoint detector. The method directly derives the relationship between the position of a target in the ground plane and the corresponding scale and position in the image and allows real-time tracking of multiple targets with high and stable degree of accuracy even at far distances and any zoom level.
Similar content being viewed by others
Notes
In the case of a PTZ sensor, the homography between each keyframe and the reference keyframe is the infinite homography \(\mathtt {H}_\infty \) that puts in relation vanishing lines and vanishing points between the images.
References
Agapito, L., Hayman, E., Reid, I.D.: Self-calibration of rotating and zooming cameras. Int. J. Comput. Vision 45(2), 107–127 (2001)
Arth, C., Klopschitz, M., Reitmayr, G., Schmalstieg, D.: Real-time self-localization from panoramic images on mobile devices. In: Proceedings of IEEE International Symposium on Mixed and Augmented Reality (2011)
Barceló, L., Binefa, X., Kender, J.R.: Robust methods and representations for soccer player tracking and collision resolution. In: Proceedings of the International Conference on Image and Video Retrieval (2005)
Bay, H., Tuytelaars, T., Gool, L.V.: SURF: speeded up robust features. In: Proceedings of the European Conference on Computer Vision (2006)
Bernardin, K., Stiefelhagen, R.: Evaluating multiple object tracking performance: the CLEAR MOT metrics. J. Image Video Process. 2008, 1–10 (2008)
Breitenstein, M., Reichlin, F., Leibe, B., Koller-Meier, E., Van Gool, L.: Online multi-person tracking-by-detection from a single, uncalibrated camera. IEEE Trans. Pattern Anal. Mach. Intell. 33(9), 1820–1833 (2011)
Brendel, W., Amer, M., Todorovic, S.: Multiobject tracking as maximum weight independent set. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (2011)
Civera, J., Davison, A.J., Magallon, J.A., Montiel, J.M.M.: Drift-free real-time sequential mosaicing. Int. J. Comput. Vision 81(2), 128–137 (2009)
Collins, R.T., Tsin, Y.: Calibration of an outdoor active camera system. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (1999)
Criminisi, A., Reid, I., Zisserman, A.: A plane measuring device. Image Vis. Comput. 17(8), 625–634 (1999)
Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2005)
Davis, J., Chen, X.: Calibrating pan-tilt cameras in wide-area surveillance networks. In: Proceedings of IEEE International Conference on Computer Vision (2003)
Del Bimbo, A., Lisanti, G., Masi, I., Pernici, F.: Device-tagged feature-based localization and mapping of wide areas with a ptz camera. In: Proceedings of CVPR Workshops, Socially Intelligent Surveillance and Monitoring (2010)
Del Bimbo, A., Lisanti, G., Masi, I., Pernici, F.: Continuous recovery for real time pan tilt zoom localization and mapping. In: Advanced Video and Signal-Based Surveillance (AVSS) (2011)
Del Bimbo, A., Lisanti, G., Pernici, F.: Scale invariant 3D multi-person tracking using a base set of bundle adjusted visual landmarks. In: Proceedings of ICCV Workshops, International Workshop on Visual Surveillance (2009)
Fischler, M.A., Bolles, R.C.: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 24(6), 381–395 (1981)
Fitzgerald, R.J.: Track biases and coalescence with probabilistic data association. IEEE Trans. Aeros. Electron. Syst. AES-21(6), 822–825 (1985). doi:10.1109/TAES.1985.310670
Hartley, R.: Self-calibration from multiple views with a rotating camera. In: Proceedings of the European Conference on Computer Vision (1994)
Hayman, E., Thorhallsson, T., Murray, D.W.: Zoom-invariant tracking using points and lines in affine views—an application of the affine multifocal tensors. In: Proceedings of the International Conference on Computer Vision (1999)
Kang, S., Paik, J.K., Koschan, A., Abidi, B.R., Abidi, M.A.: Real-time video tracking using PTZ cameras. In: Proceedings of International Conference on Quality Control by Artificial Vision (2003)
Klein, G., Murray, D.: Parallel tracking and mapping for small AR workspaces. In: Proceedings of the IEEE and ACM International Symposium on Mixed and Augmented Reality (2007)
Leibe, B., Leonardis, A., Schiele, B.: Robust object detection with interleaved categorization and segmentation. Int. J. Comput. Vision 77(1), 259–289 (2007)
Liebowitz, D., Zisserman, A.: Metric rectification for perspective images of planes. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (1998)
Lim, H., Sinha, S., Cohen, M., Uyttendaele, M., Kim, H.J.: Real-time monocular image-based 6-dof localization. Int. J. Robot. Res.: IJRR 34(4–5), 476–4925 (2015)
Lovegrove, S., Davison, A.J.: Real-time spherical mosaicing using whole image alignment. In: Proceedings of European Conference on Computer Vision (2010)
Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60(2), 91–110 (2004)
Okuma, K., Taleghani, A., de Freitas, N., Little, J.J., Lowe, D.G.: A boosted particle filter: multitarget detection and tracking. In: Proceedings of the European Conference on Computer Vision (2004)
Pernici, F., Del Bimbo, A.: Object tracking by oversampling local features. IEEE Trans. Pattern Anal. Mach. Intell. 36(12), 2538–2551 (2014)
Seo, Y., Choi, S., Kim, H., Hong, K.S.: Where are the ball and players? Soccer game analysis with color based tracking and image mosaick. In: Proceedings of the International Conference on Image Analysis and Processing (1997)
Sinha, S., Pollefeys, M.: Towards calibrating a pan-tilt-zoom cameras network. In: Proceedings of ECCV Workshops, Omnidirectional Vision and Camera Networks (2004)
Sinha, S., Pollefeys, M.: Pan-tilt-zoom camera calibration and high-resolution mosaic generation. Comput. Vis. Image Underst. 103(3), 170–183 (2006)
Song, D., Goldberg, K.: A minimum variance calibration algorithm for pan-tilt robotic cameras in natural environments. In: IEEE International Conference on Robotics and Automation (2006)
Tao, H., Sawhney, H.S., Kumar, R.: Object tracking with bayesian estimation of dynamic layer representations. IEEE Trans. Pattern Anal. Mach. Intell. 24(1), 75–89 (2002)
Tordoff, B., Murray, D.: Reactive control of zoom while fixating using perspective and affine cameras. IEEE Trans. Pattern Anal. Mach. Intell. 26(1), 98–112 (2004)
Varcheie, P., Bilodeau, G.A.: Active people tracking by a PTZ camera in ip surveillance system. In: IEEE International Workshop on Robotic and Sensors Environments (2009)
Williams, B., Klein, G., Reid, I.: Real-time SLAM relocalisation. In: Proceedings of the IEEE International Conference on Computer Vision (2007)
Wu, Z., Radke, R.: Keeping a pan-tilt-zoom camera calibrated. IEEE Trans. Pattern Anal. Mach. Intell. 35(8), 1994–2007 (2013)
Yang, B., Huang, C., Nevatia, R.: Learning affinities and dependencies for multi-target tracking using a CRF model. In: Proceedings of IEEE Conference on Computer Vision and Patter Recognition (2011)
Cai, Y.,de Freitas, N., Little, J.: Robust visual tracking for multiple targets. In: Proceedings of the European Conference on Computer Vision (2006)
Acknowledgments
This work is partially supported by THALES Italia Spa, Florence, Italy.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Lisanti, G., Masi, I., Pernici, F. et al. Continuous localization and mapping of a pan–tilt–zoom camera for wide area tracking. Machine Vision and Applications 27, 1071–1085 (2016). https://doi.org/10.1007/s00138-016-0799-x
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00138-016-0799-x