Learning Object Appearance from Occlusions Using Structure and Motion Recovery

  • Kai Cordes
  • Björn Scheuermann
  • Bodo Rosenhahn
  • Jörn Ostermann
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7726)


Visual effect creation as used in movie production often require structure and motion recovery and video segmentation. Both techniques are essential to integrate virtual objects between scene elements. In this paper, a new method for video segmentation is presented. It incorporates 3D scene information from the structure and motion recovery. By connecting and evaluating discontinued feature tracks, occlusion and reappearance information is obtained during sequential camera and scene estimation.

The foreground is characterized as image regions which temporarily occlude the rigid scene structure. The scene structure is represented by reconstructed object points. Their projections onto the camera images provide the cues for regions classified as foreground or background. The knowledge of occluded parts of a connected feature track is used to feed the object segmentation which crops the foreground image regions automatically.

Two applications are presented: the occlusion of integrated virtual objects and the blurred background effect. Several demonstrations on official and self-made data show very realistic results in augmented reality.


Gaussian Mixture Model Virtual Object Object Point Foreground Object Bundle Adjustment 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Pollefeys, M., Gool, L.V.V., Vergauwen, M., Verbiest, F., Cornelis, K., Tops, J., Koch, R.: Visual modeling with a hand-held camera. International Journal of Computer Vision (IJCV) 59, 207–232 (2004)CrossRefGoogle Scholar
  2. 2.
    Zhang, G., Dong, Z., Jia, J., Wong, T.-T., Bao, H.: Efficient Non-consecutive Feature Tracking for Structure-from-Motion. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010, Part V. LNCS, vol. 6315, pp. 422–435. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  3. 3.
    Cordes, K., Müller, O., Rosenhahn, B., Ostermann, J.: Feature Trajectory Retrieval with Application to Accurate Structure and Motion Recovery. In: Bebis, G. (ed.) ISVC 2011, Part I. LNCS, vol. 6938, pp. 156–167. Springer, Heidelberg (2011)Google Scholar
  4. 4.
    Hillman, P., Lewis, J., Sylwan, S., Winquist, E.: Issues in adapting research algorithms to stereoscopic visual effects. In: IEEE International Conference on Image Processing (ICIP), pp. 17–20 (2010)Google Scholar
  5. 5.
    Sand, P., Teller, S.: Particle video: Long-range motion estimation using point trajectories. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), vol. 2, pp. 2195–2202 (2006)Google Scholar
  6. 6.
    Apostoloff, N.E., Fitzgibbon, A.W.: Automatic video segmentation using spatiotemporal t-junctions. In: British Machine Vision Conference, BMVC (2006)Google Scholar
  7. 7.
    Brox, T., Malik, J.: Object Segmentation by Long Term Analysis of Point Trajectories. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010, Part V. LNCS, vol. 6315, pp. 282–295. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  8. 8.
    Sheikh, Y., Javed, O., Kanade, T.: Background subtraction for freely moving cameras. In: IEEE International Conference on Computer Vision (ICCV), pp. 1219–1225 (2009)Google Scholar
  9. 9.
    Zhang, G., Jia, J., Hua, W., Bao, H.: Robust bilayer segmentation and motion/depth estimation with a handheld camera. IEEE Transaction on Pattern Analysis and Machine Intelligence (PAMI) 33, 603–617 (2011)zbMATHCrossRefGoogle Scholar
  10. 10.
    Boykov, Y., Jolly, M.P.: Interactive graph cuts for optimal boundary & region segmentation of objects in n-d images. In: IEEE International Conference on Computer Vision (ICCV), vol. 1, pp. 105–112 (2001)Google Scholar
  11. 11.
    Triggs, B., McLauchlan, P.F., Hartley, R.I., Fitzgibbon, A.W.: Bundle adjustment - a modern synthesis. In: Proceedings of the International Workshop on Vision Algorithms: Theory and Practice, IEEE International Conference on Computer Vision (ICCV), pp. 298–372. Springer (2000)Google Scholar
  12. 12.
    Hartley, R.I., Zisserman, A.: Multiple View Geometry, 2nd edn. Cambridge University Press (2003)Google Scholar
  13. 13.
    Lucas, B., Kanade, T.: An iterative image registration technique with an application to stereo vision. In: International Joint Conference on Artificial Intelligence (IJCAI), pp. 674–679 (1981)Google Scholar
  14. 14.
    Lowe, D.G.: Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision (IJCV) 60, 91–110 (2004)CrossRefGoogle Scholar
  15. 15.
    Fischler, R.M.A., Bolles, C.: Random sample consensus: A paradigm for model fitting with application to image analysis and automated cartography. Communications of the ACM 24, 381–395 (1981)MathSciNetCrossRefGoogle Scholar
  16. 16.
    Cordes, K., Scheuermann, B., Rosenhahn, B., Ostermann, J.: Occlusion handling for the integration of virtual objects into video. In: Csurka, G., Braz, J. (eds.) International Conference on Computer Vision Theory and Applications (VISAPP), pp. 173–180. SciTePress (2012)Google Scholar
  17. 17.
    Rother, C., Kolmogorov, V., Blake, A.: Grabcut: interactive foreground extraction using iterated graph cuts. ACM SIGGRAPH Papers 23, 309–314 (2004)CrossRefGoogle Scholar
  18. 18.
    Thormählen, T., Hasler, N., Wand, M., Seidel, H.P.: Registration of sub-sequence and multi-camera reconstructions for camera motion estimation. Journal of Virtual Reality and Broadcasting 7 (2010)Google Scholar
  19. 19.
    Scheuermann, B., Rosenhahn, B.: SlimCuts: GraphCuts for High Resolution Images Using Graph Reduction. In: Boykov, Y., Kahl, F., Lempitsky, V., Schmidt, F.R. (eds.) EMMCVPR 2011. LNCS, vol. 6819, pp. 219–232. Springer, Heidelberg (2011)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Kai Cordes
    • 1
  • Björn Scheuermann
    • 1
  • Bodo Rosenhahn
    • 1
  • Jörn Ostermann
    • 1
  1. 1.Institut für Informationsverarbeitung (TNT)Leibniz Universität HannoverGermany

Personalised recommendations