Journal of Real-Time Image Processing

, Volume 5, Issue 1, pp 57–70 | Cite as

Aligning windows of live video from an imprecise pan-tilt-zoom camera into a remote panoramic display for remote nature observation

  • Dezhen SongEmail author
  • Yiliang Xu
  • Ni Qin
Original Research Paper


A pan-tilt-zoom (PTZ) robotic camera can provide a detailed live video of selected areas of interest within a large potential viewing field. The selective coverage is ideal for nature observation applications where power and bandwidth are often limited. To provide the spatial context for human observers, it is desirable to insert the live video into a large spherical panoramic display representing the entire viewing field. Accurate alignment of the video stream within the panoramic display is difficult due to imprecise pan-tilt values and rapid changes in camera configurations. Common image alignment algorithms are computationally expensive for real time applications. We are interested in designing algorithms that fit low power computation platform and hence can be implemented inside the PTZ camera in the future. We present a sampling-based constant-time image alignment algorithm based on spherical projection and projection-invariant selective sampling that accurately registers paired images at 100 frames per second on a simulated embedded platform. The alignment accuracy actually is better than existing methods when high rotational difference is involved. Experiments suggest that the new alignment algorithm is faster than existing algorithms by 1,471.6 times when aligning a six-mega-pixel image pair.


Image alignment Pan-tilt-zoom camera Panorama video Motion panorama Nature observation 



Thanks to J. Zhang for his help in implementing part of the algorithms. Thanks are given to H. Lee, C. Kim, and Z. Bing for their contributions to NetBot Laboratory, Department of Computer Science and Engineering, Texas A&M University. Thanks to K. Goldberg, J. Yi, D. Volz, R. Gutierrez-Osuna, and V. Taylor for insightful discussions and feedback.


  1. 1.
    Benosman, R., Kang, S.B.: Panoramic vision. Springer, New York (2001)zbMATHGoogle Scholar
  2. 2.
    Foote, J., Kimber, D.: Flycam: practical panoramic video and automatic camera control. In: IEEE International Conference on Multimedia and Expo, vol. 3. ICME 2000, New York, pp. 1419–1422 (2000)Google Scholar
  3. 3.
    Foote, J., Kimber, D.: Enhancing distance learning with panoramic video. In: Proceedings of the 34th Hawaii International Conference on System Sciences (2001)Google Scholar
  4. 4.
    Song D., Goldberg K.: Approximate algorithms for a collaboratively controlled robotic camera. IEEE. Trans. Robotics 23(5), 1061–1070 (2007) CrossRefGoogle Scholar
  5. 5.
    Song, D.: Sharing a Vision: Systems and Algorithms for Collaboratively-Teleoperated Robotic Cameras. Springer, New York (2009)Google Scholar
  6. 6.
    Song D., Qin N., Goldberg K.: Systems, control models, and codec for collaborative observation of remote environments with an autonomous networked robotic camera. Auton. Robots 24(4), 435–449 (2008)CrossRefGoogle Scholar
  7. 7.
    Song, D., Qin, N., Goldberg, K.: A minimum variance calibration algorithm for pan-tilt robotic cameras in natural environments. In: IEEE International Conference on Robotics and Automation (ICRA), Orlando (2006)Google Scholar
  8. 8.
    Swaminathan R., Nayar S.K.: Nonmetric calibration of wide-angle lenses and polycameras. IEEE Trans. Pattern Anal. Mach. Intell. 22(10), 1172–1178 (2000)CrossRefGoogle Scholar
  9. 9.
    Tan K.-H., Hua H., Ahuja N.: Multiview panoramic cameras using mirror pyramids. IEEE Trans. Pattern Anal. Mach. Intell. 26(7), 1941–946 (2004)Google Scholar
  10. 10.
    Baker S., Nayar S.K.: A theory of single-viewpoint catadioptric image formation. Int. J. Comp. Vis. 35(2), 175–196 (1999)CrossRefGoogle Scholar
  11. 11.
    Nayar, S.K.: Catadioptric omnidirectional camera. In: Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, San Juan, Puerto Rico, pp. 482–488 (1997)Google Scholar
  12. 12.
    Xiong, Y., Turkowski, K.: Creating image-based vr using a self-calibrating fisheye lens. In: Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, San Juan, Puerto Rico, pp. 237–243 (1997)Google Scholar
  13. 13.
    Szeliski, R.: Image alignment and stitching, microsoft research, technical report msr-tr-2004-92 (2004)Google Scholar
  14. 14.
    Kim, B.Y., Jang, K.H., Jung, S.K.: Adaptive strip compression for panorama video streaming. In Computer Graphics International (CGI’04), Crete, Greece (2004)Google Scholar
  15. 15.
    Szeliski, R.: Image alignment and stitching: a tutorial. Found. Trends. Comput. Graph. Vis. 2(1), 1–104 (2006)CrossRefGoogle Scholar
  16. 16.
    Shum, H.-Y., Szeliski, R.: Panoramic image mosaics. Microsoft Research Technical Report MSR-TR-97-23, Tech. Rep. (1997)Google Scholar
  17. 17.
    Hartley R.: Self-calibration of stationary cameras. Int. J. Comp. Vis. 22, 5–23 (1997) CrossRefMathSciNetGoogle Scholar
  18. 18.
    Coorg S., Teller S.: Spherical mosaics with quaternions and dense correlation. Int. J. Comp. Vis. 37(3), 259–273 (2000)zbMATHCrossRefGoogle Scholar
  19. 19.
    Castro E.D., Morandi C.: Registration of translated and rotated images using finite fourier transform. IEEE Trans. Pattern Anal. Mach. Intell. 9(5), 700–703 (1987)CrossRefGoogle Scholar
  20. 20.
    Reddy B.S., Chatterji B.N.: An fft-based technique for translation, rotation, and scale-invariant image registration. IEEE Trans. Image Process 5(8), 1266–1271 (1996)CrossRefGoogle Scholar
  21. 21.
    Hu, B., Brown, C., Choi, A.: Acquiring an environment map through image mosaicking. In University of Rochester. TR-786, November 2001Google Scholar
  22. 22.
    Cho, S.-H., Chung, Y.-K., Lee, J.Y.: Automatic image mosaic system using image feature detection and taylor series. In Proceedings of VIIth Digital Image Computing: Techniques and Applications. Sydney, pp. 549–556 (2003)Google Scholar
  23. 23.
    Kanazawa, Y., Kanatani, K.: Image mosaicing by stratified matching. In: Proceedings of Statistical Methods in Video Processing Workshop, Denmark, pp. 31–36 (2002)Google Scholar
  24. 24.
    Zhang, W., Kosecka, J., Li, F.: Mosaics construction from a sparse set of views. In: Proceedings of First International Symposium on 3D Data Processing Visualization and Transmission, pp. 177–180 (2002)Google Scholar
  25. 25.
    Harris C.J., Stephens M.: A combined corner and edge detector. In: Proceedings 4th Alvey Vision Conference, Manchester, pp. 147–151 (1988)Google Scholar
  26. 26.
    Torr, P.H.S., Zisserman, A.: Feature based methods for structure and motion estimation. In: Proceedings of the International Workshop on Vision Algorithm: Theory and Practice, Corfu, pp. 278–294 (1999)Google Scholar
  27. 27.
    Zoghlami, I., Faugeras, O., Deriche, R.: Using geometric corners to build a 2d mosaic from a set of images. In: Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition, San Juan, Puerto Rico, pp. 420–425 (1997)Google Scholar
  28. 28.
    Brown M., Lowe D. (2003) Recognising panoramas. In: Proceedings of IEEE International Conference on Computer Vision, Nice, France, vol. 2, pp. 1218–1225Google Scholar
  29. 29.
    Bay, H., Tuytelaars, T., Gool, L.V.: Surf: Speeded up robust features. In: 9th European Conference on Computer Vision (ECCV), Graz, Austria, pp. 404–417 (2006)Google Scholar
  30. 30.
    Zhang Z., Deriche R., Faugeras O., Luong Q.T.: A robust technique for matching two uncalibrated images through the recovery of the unknown epipolar geometry. Art. Intell. 78, 87–119 (1995)CrossRefGoogle Scholar
  31. 31.
    Triggs B., McLauchlan P.F., Hartley R.I., Fitzgibbon A.W.: Bundle adjustment—a modern synthesis. In ICCV: Proceedings of the International Workshop on Vision Algorithms. Springer, London, pp. 298–372 (2000)Google Scholar
  32. 32.
    Steedly, D., Pal, C., Szeliski, R.: Efficiently registering video into panoramic mosaics. In: Proceedings of Tenth IEEE International Conference on Computer Vision, vol. 2, pp. 1300–1307 (2005)Google Scholar
  33. 33.
    Lee S., Hong M., Wee J. (2005) Low-hardware-cost motion estimation with large search range for vlsi multimedia processors. IEICE Trans. Inf. Syst. E88-D(9), 2177–2182CrossRefGoogle Scholar
  34. 34.
    Babionitakis K., Doumenis G., Georgakarakos G., Lentaris G., Nakos K., Reisis D., Sifnaios I., Vlassopoulos N.: A real-time motion estimation FPGA architecture. J. RealTime Image Proc. 3(1–2), 3–20 (2008)CrossRefGoogle Scholar
  35. 35.
    Mazare, S., Dugelay, J., Pacalet, R.: Using gpu for fast block-matching. In: European Signal Processing Conference, Florence, Italy (2006)Google Scholar
  36. 36.
    Trucco E., Doull A., Odone F., Fusiello A., Lane D.: Dynamic video mosaicing and augmented reality for subsea inspection and monitoring. Oceanology International Conference, Brighton, UK (2000)Google Scholar
  37. 37.
    Agarwala, A., Zheng, C., Pal, C., Agrawala, M., Cohen, M., Curless, B., Salesin, D., Szeliski, R.: Panoramic video textures. In: ACM Transactions on Graphics (Proceedings of SIGGRAPH 2005), Los Angeles (2005)Google Scholar
  38. 38.
    Hartley, R., Zisserman, A.: Multiple View Geometry in Computer Vision. Cambridge Press, London (2000)Google Scholar
  39. 39.
    Gonzalez, R., Woods, R.: Digital Image Processing. Addison-Wesley Publishing Company, USA (1992)Google Scholar
  40. 40.
    Jolliffe, I.: Principal Component Analysis. Springer, New York (2002)Google Scholar
  41. 41.
    Stockman G., Shapiro L.G.: Computer Vision. Prentice Hall, Upper Saddle River (2001)Google Scholar

Copyright information

© Springer-Verlag 2009

Authors and Affiliations

  1. 1.Computer Science and Engineering DepartmentTexas A&M UniversityCollege StationUSA

Personalised recommendations