Journal of Real-Time Image Processing

, Volume 2, Issue 2–3, pp 81–101 | Cite as

Extraction of 3D freeform surfaces as visual landmarks for real-time tracking

  • Bogumil Bartczak
  • Kevin Koeser
  • Felix Woelk
  • Reinhard Koch
Special Issue


This work presents a system for the generation of a free-form surface model from video sequences. Although any single centered camera can be applied in the proposed system the approach is demonstrated using fish-eye lenses because of their good properties for tracking. The system is designed to function automatically and to be flexible with respect to size and shape of the reconstructed scene. To minimize geometric assumptions a statistic fusion of dense depth maps is utilized. Special attention is payed to the necessary rectification of the spherical images and the resulting iso-disparity surfaces, which can be exploited in the fusion approach. Before dense depth estimation can be performed the cameras’ pose parameters are extracted by means of a structure-from-motion (SfM) scheme. In this respect automation of the system is achieved by thorough decision model based on robust statistics and error propagation of projective measurement uncertainties. This leads to a scene-independent set of only a few parameters. All system components are formulated in a general way, making it possible to cope with any single centered projection model, in particular with spherical cameras. In using wide field-of-view cameras the presented system is able to reliably retrieve poses and consistently reconstruct large scenes. A textured triangle mesh constructed on basis of the scene’s reconstructed depth, makes the system’s results suitable to function as reference models in a GPU driven analysis-by-synthesis framework for real-time tracking.


Omnidirectional vision Visual scene reconstruction Rectification Robust statistics Structure-from-Motion 



This work has been partially funded by the European Union in project MATRIS IST-002013.


  1. 1.
    Adelson, E.H., Bergen, J.: The plenoptic function and the elements of early vision. In: Landy, M., Movshon, J.A. (eds.) Computation Models of Visual Processing, pp. 3–20. MIT Press, Cambridge (1991)Google Scholar
  2. 2.
    Arican, Z., Frossard, P.: Dense disparity estimation from omnidirectional images. In: Proceedings of the 2007 IEEE International Conference on Advanced Video and Signal based Surveillance (2007)Google Scholar
  3. 3.
    Ayache, N., Hansen, C.: Rectification of images for binocular and trinocular stereovision. In: Proceedings International Conference on Pattern Recognition, pp. 11–16 (1988)Google Scholar
  4. 4.
    Badri, J., Tilmant, C., Lavest, J.M., Pham, Q.C., Sayd, P.: Camera-to-camera mapping for hybrid pan-tilt-zoom sensors calibration, pp. 132–141 (2007)Google Scholar
  5. 5.
    Baker, S., Matthews, I.: Lucas-Kanade 20 Years On: A Unifying Framework. Int. J. Comput. Vis. 56(3), 221–255 (2004)CrossRefGoogle Scholar
  6. 6.
    Peter, B., Sven, F., Wolfgang, S.: The Wägele: a mobile platform for acquisition of 3D models of indoor outdoor environments. In: 9th Tübingen Perception Conference (TWK 2006), Tübingen (2006)Google Scholar
  7. 7.
    Birchfield, S., Tomasi, C.: Depth discontinuities by pixel-to-pixel stereo. Int. J. Comput. Vis. (1999)Google Scholar
  8. 8.
    Beder, C., Steffen, R.: Determining an initial image pair for fixing the scale of a 3D reconstruction from an image sequence. In: Franke, K., Müller, K-R., Nickolay, B., Schäfer, R. (eds.) Pattern Recognition, number 4174 in LNCS, pp. 657–666. Springer, Heidelberg (2006)Google Scholar
  9. 9.
    Cornelius, H., Sara, R., Martinec, D., Pajdla, T., Chum, O., Matas, J.: Towards complete free-form reconstruction of complex 3D scenes from an unordered set of uncalibrated images. In: Statistical Methods in Video Processing, ECCV 2004 Workshop, pp. 1–12. Springer, Heidelberg (2004)Google Scholar
  10. 10.
    Falkenhagen, L.: Hierarchical block-based disparity estimation considering neighbourhood constraints. In: International Workshop on SNHC and 3D Imaging. Rhodes, Greece (1997)Google Scholar
  11. 11.
    Fischler, M., Bolles, R.: RANdom SAmpling Consensus: a paradigm for model fitting with application to image analysis and automated cartography. Commun. ACM 24(6), 381–395 (1981)CrossRefMathSciNetGoogle Scholar
  12. 12.
    Fleck, M.: Perspective projection: the wrong imaging model. Technical Report TR 95-01, Computer Science, University of Iowa (1995)Google Scholar
  13. 13.
    Förstner, W.: Uncertainty and projective geometry. In: Handbook of Computational Geometry for Pattern Recognition, Computer Vision, Neurocomputing and Robotics, pp. 493–534. Springer, Heidelberg (2005)Google Scholar
  14. 14.
    Fusiello, A.: Improving feature tracking with robust statistics. In: Pattern Analysis and Application. Springer, London 2:312–320 (1999)Google Scholar
  15. 15.
    Geyer, C., Daniilidis, K.: Catadioptric projective geometry. Int. J. Comput. Vis. 43, 223–243 (2001)CrossRefGoogle Scholar
  16. 16.
    Geyer C., Daniilidis, K.: (2003) Conformal rectification of omnidirectional stereo Pairs. In: Proceedings of Omnivis 2003: Workshop on Omnidirectional Vision and Camera NetworksGoogle Scholar
  17. 17.
    Gonzalez-Barbosa, J-J., Lacroix, S.: (2005) Fast dense panoramic stereovision. In: Proceedings of the 2005 IEEE International Conference on Robotics and Automation, pp. 1210–1215Google Scholar
  18. 18.
    Hampel, F.R., Ronchetti, E.M., Rousseeuw, P.J., Stahel, W.A.: Robust statistics: the approach based on influence functions (Wiley Series in Probability and Statistics), revised edition. Wiley, New York (2005)Google Scholar
  19. 19.
    Hartley, R.: Theory and practice of projective rectification. Int. J. Comput. Vis. 35(2), 115–127 (1999)CrossRefMathSciNetGoogle Scholar
  20. 20.
    Hartley, R., Zisserman, A.: Multiple View Geometry in Computer Vision. Cambridge University Press, Cambridge (2000)Google Scholar
  21. 21.
    Hartley, R., Zissermann, A.: Multiple View Geometry in Computer Vision, second edition. Cambridge University Press, Cambridge (2004)Google Scholar
  22. 22.
    Aloimonos, Y., Neumann, J., Fermuller, C.: Eyes from eyes: new cameras for structure from motion. In: In IEEE Workshop on Omnidirectional Vision, pp. 19–26 (2002)Google Scholar
  23. 23.
    McGlone, J.C.: Manual of Photogrammetry, 5th edition. ASPRS (2004)Google Scholar
  24. 24.
    Julier, S.J., Uhlmann, J.K., Durrant-Whyte, H.F.: New Approach for Filtering Nonlinear Systems. In: Proceedings of the American Control Conference (ACC), pp. 1628–1632. Seattle WA, USA (1995)Google Scholar
  25. 25.
    Julier, S.J., Uhlmann, J.K.: Reduced sigma point filters for the propagation of means and covariances through nonlinear transformations. In: Proceedings of the IEEE American Control Conference, pp. 887–892, Anchorage AK, USA, 8–10 May (2002). IEEE (2002)Google Scholar
  26. 26.
    Julier, S.J., Uhlmann, J.K.: The scaled unscented transformation. In: Proceedings of the IEEE American Control Conference, pp. 4555–4559. Anchorage AK, USA, 8–10 May (2002). IEEE (2002)Google Scholar
  27. 27.
    Kenichi K.: Statistical optimization for geometric computation: theory and practice. Dover, New York (2005)Google Scholar
  28. 28.
    Kang, S.B., Szeliski, R.: 3-D scene data recovery using omnidirectional multibaseline stereo. In: CVPR ’96: Proceedings of the 1996 Conference on Computer Vision and Pattern Recognition (CVPR ’96), pp. 364. IEEE Computer Society, Washington, DC, USA (1996)Google Scholar
  29. 29.
    Koch, R., Koeser, K., Streckel, B., Evers-Senne, J-F.: Markerless Image-based 3D tracking for real-time augmented reality applications. In: WIAMIS 2005. Montreux, Switzerland (2005)Google Scholar
  30. 30.
    Koeser, K., Bartczak, B., Koch, R.: Drift-free pose estimation with hemispherical cameras. In: Proceedings of CVMP 2006, pp. 20–28. London (2006)Google Scholar
  31. 31.
    Koeser, K., Bartczak, B., Koch, R.: Robust GPU-assisted camera tracking using free-form surface models. J Real-Time Image Proc (submitted to same issue) (2007)Google Scholar
  32. 32.
    Lepetit, V., Fua, P.: Monocular model-based 3D tracking of rigid objects: a survey. Found. Trends Comput. Graph. Vis. 1(1):1–89 (2005)CrossRefGoogle Scholar
  33. 33.
    Lepetit, V., Lagger, P., Fua, P.: Randomized trees for real-time keypoint recognition. In: Conference on Computer Vision and Pattern Recognition 2, 775–781 (2005)Google Scholar
  34. 34.
    Li, S.: Real-time spherical stereo. In: ICPR ’06: Proceedings of the 18th International Conference on Pattern Recognition, pp. 1046–1049. IEEE Computer Society, Washington, DC, USA (2006)Google Scholar
  35. 35.
    Loop, C., Zhang, Z.: Computing rectifying homographies for stereo vision. Proc. Comput. Vis. Pattern Recogn. 125–131 (1999)Google Scholar
  36. 36.
    Lucas, B.D., Kanade, T.: An iterative image registration technique with an application to stereo vision. In: IJCAI81, pp. 674–679 (1999)Google Scholar
  37. 37.
    Van Meerbergen, G., Vergauwen, M., Pollefeys, M., Van Gool, L.: A hierarchical symmetric stereo algorithm using dynamic programming. Int. J. Comput. Vis. 47(1–3), 275–285 (2002)MATHCrossRefGoogle Scholar
  38. 38.
    Meguro, J., Takiguchi, J., Amano, Y., Hashizume, T.: 3D reconstruction using multibaseline omnidirectional motion stereo based on gps/dead-reckoning compound navigation system. Int. J. Robot. Res. 26(6), 625–636 (2007)CrossRefGoogle Scholar
  39. 39.
    Micusik, B.: Two-view geometry of omnidirectional cameras. PhD thesis, Czech Technical University in Prague, Technicka 2, 166 27 Prague 6, Czech Republic (2004)Google Scholar
  40. 40.
    Micusik, B., Pajdla, P.: Estimation of omnidirectional camera model from epipolar geometry. In: Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR’03), Vol. 1, p. 485. IEEE Computer Society (2003)Google Scholar
  41. 41.
    Micusik, B., Pajdla, T.: Structure from Motion with Wide Circular Field of View Cameras. IEEE Trans. Pattern Anal. Mach. Intell. 28(7), 1135–1149 (2006)CrossRefGoogle Scholar
  42. 42.
    Mikhail, E.M., Ackermann, F.E.: Observations and least squares. IEP, New York (1976)Google Scholar
  43. 43.
    Mordohai, P., Frahm, J-M., Akbarzadeh, A., Clipp, B., Engels, C., Gallup, D., Merrell, P., Salmi, C., Sinha, S., Talton, B., Wang, L., Yang, Q., StewTnius, H., Towles, H., Welch, G., Yang, R., Pollefeys, M., NistTr, D.: Real-time video-based reconstruction of urban environments. In: 3D-ARCH 2007: 3D Virtual Reconstruction and Visualization of Complex Architectures, Zurich, Switzerland (2007)Google Scholar
  44. 44.
    Nistér, D.: Automatic dense reconstruction from uncalibrated video sequences. PhD thesis, Kungl Tekniska Hogskolen (2001)Google Scholar
  45. 45.
    Nistér, D.: Preemptive RANSAC for live structure and motion estimation. In: Proceedings of the ICCV, pp. 199–206 (2003)Google Scholar
  46. 46.
    Nistér, D.: An efficient solution to the five-point relative pose problem. TPAMI (2004)Google Scholar
  47. 47.
    Pajarola, R., Meng, Y., Sainz, M.: Fast depth-image meshing and warping. Technical Report UCI-ECE-TR-02-02, The Henry Samueli School of Engineering, University of California, Irvine (2002)Google Scholar
  48. 48.
    Perwass, C., Sommer, G.: The inversion camera model. In: 28. Symposium für Mustererkennung, DAGM 2006, Berlin, 12–14.09.2006, number 4174 in LNCS, pp. 647–656. Springer, Heidelberg (2006)Google Scholar
  49. 49.
    Pollefeys, M., Koch, R., Van Gool, L.: A simple and efficient rectification method for general motion. In: ICCV (1), pp. 496–501 (1999)Google Scholar
  50. 50.
    Pollefeys, M., Sinha, S.: Iso-disparity Surfaces for general Stereo Configurations. In: Pajdla, T., Matas, J. (eds.) Computer vision - ECCV 2004, Vol. 3023, pp. 509–520. Springer, Heidelberg (2004)Google Scholar
  51. 51.
    Pollefeys, M., van Gool, L., Vergauwen, M., Verbiest, F., Cornelis, K., Tops, J., Reinhard, K.: Visual modeling with a hand-held camera. Int. J. Comput. Vis. 59(3), 207–232 (2004)CrossRefGoogle Scholar
  52. 52.
    Roy, S., Meunier, J., Cox, I.: Cylindrical rectification to minimize epipolar distortion (1997)Google Scholar
  53. 53.
    Scaramuzza, D., Martinelli, A., Siegwart, R.: A flexible technique for accurate omnidirectional camera calibration and structure from motion. In: Proceedings of IEEE International Conference of Vision Systems. IEEE (2006)Google Scholar
  54. 54.
    Scharstein, D., Szeliski, R., Zabih, R.: A Taxonomy and evaluation of dense two-frame stereo correspondence algorithms. In: Proceedings of IEEE Workshop on Stereo and Multi-BaselineVision. Kauai, HI (2001)Google Scholar
  55. 55.
    Schroeder, W.J., Zarge, J.A., Lorensen, W.E.: Decimation of triangle meshes. Comput. Graph. 26(2), 65–70 (1992)CrossRefGoogle Scholar
  56. 56.
    Seitz, S., Curless, B., Diebel, J., Scharstein, D., Szeliski, R.: A comparison and evaluation of multi-view stereo reconstruction algorithms (2006)Google Scholar
  57. 57.
    Shi, J., Tomasi, C.: Good features to track. In: Conference on computer vision and pattern recognition. Seattle, IEEE, pp. 593–600 (1994)Google Scholar
  58. 58.
    Skrypnyk, I., Lowe, D.G.: Scene modelling, recognition and tracking with invariant image features. In: IEEE and ACM International Symposium on Mixed and Augmented Reality, pp. 110–119 (2004)Google Scholar
  59. 59.
    Streckel, B., Koch, R.: Lens model selection for visual tracking. In: Lecture Notes in Computer Science 3663 (DAGM 2005), Vienna, Austria (2005)Google Scholar
  60. 60.
    Takiguchi, J.-I., Yoshida, M., Takeya, A., Eino, J-I., Hashizume, T.: High precision range estimation from an omnidirectional stereo system. In: Jacques, G.V. (eds.) Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems 1, 263–268 (2002)Google Scholar
  61. 61.
    Thomas, G.A., Jin, J., Niblett, T., Urquhart, C.: A versatile camera position measurement system for virtual reality. In: Proceedings of International Broadcasting Convention, pp. 284–289 (1997)Google Scholar
  62. 62.
    Triggs, B., McLauchlan, P., Hartley, R., Fitzgibbon, A.: Bundle adjustment—a modern synthesis. In: Triggs, B, Zisserman, A, Szeliski, R (eds.) Vision Algorithms: Theory and Practice, Vol. 1883 of Lecture Notes in Computer Science. Springer, Heidelberg, pp. 298–372 (2000)Google Scholar
  63. 63.
    Zhang, Z.Y.: Parameter-estimation techniques: a tutorial with application to conic fitting. IVC 15(1), 59–76 (1997)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag 2007

Authors and Affiliations

  • Bogumil Bartczak
    • 1
  • Kevin Koeser
    • 1
  • Felix Woelk
    • 1
  • Reinhard Koch
    • 1
  1. 1.Institute of Computer ScienceChristian-Albrechts-Universität KielKielGermany

Personalised recommendations