Skip to main content
SpringerLink
Log in

AVScreen: a real-time video augmentation method

  • Special Issue Paper
  • Published:
Journal of Real-Time Image Processing Aims and scope Submit manuscript

Abstract

We present a tool for video augmentation in real-time, which we name the augmentation virtual screen (AVScreen). AVScreen is useful for developing advertisements, commercials, music videos, movies, etc. The main challenges for augmenting videos, in contrast to fixed images, is that moving objects in the foreground may occlude the region to be augmented in the background and that the composition can be affected by camera movements. Therefore, we use a procedure for foreground–background video segmentation in order to deal with such occlusions. Comparisons with foreground–background video segmentation methods of the state of the art in both accuracy and computational efficiency support our choice: we reduce around 70 % of the segmentation error in a popular benchmark database and achieve real-time performance. Moreover, a new stabilization method to augment unstable camera videos is presented. For augmenting video shots, we present an efficient graph-based method for panorama (mosaic) computation. The real-time performance is reached by implementing high computational demanding procedures in GPU. The frame rate of our method is 18 frames per second for a video size of 640 × 480 pixels.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Notes

  1. Corner detection function “goodFeaturesToTrack” in the OpenCV library [32].

  2. Lucas-Kanade’s optical flow method implemented in the OpenCV library (function “calcOpticalFlowPyrLK”).

  3. RANSAC method for homography calculation is implemented in the OpenCV library (function “findHomography”).

  4. A real-time implementation of the Graph Cut algorithm is available in the NVIDIA NPP library.

References

  1. Ahn, J.H., Byun, H.: Accurate foreground extraction using graph cut with trimap estimation. In: Chang, L.W., Lie, W.N. (eds.) Advances in image and video technology, lecture notes in Computer Science, vol. 4319, pp. 1185–1194. Springer Berlin, Heidelberg (2006)

  2. Chen, X., Li, Q., Li, X., Zhao, Q.: Video motion stitching using trajectory and position similarities. SCIENCE CHINA Inf. Sci. 55(3):600–614 (2012)

    Article  MathSciNet  Google Scholar 

  3. Colombari, A., Fusiello, A., Murino, V.: Segmentation and tracking of multiple video objects. Pattern Recogn. 40(4):1307–1317 (2007)

    Article  MATH  Google Scholar 

  4. Douze, M., Charvillat, V.: Real-time generation of augmented video sequences by background tracking: research articles. Comput. Animat. Virtual Worlds 17(5):537–550 (2006)

    Article  Google Scholar 

  5. Fernández, C., Baiget, P., Roca, F.X., Gonzààlez, J.: Augmenting video surveillance footage with virtual agents for incremental event evaluation. Pattern Recogn. Lett. 32(6):878–889 (2011)

    Article  Google Scholar 

  6. Geman, S., Geman, D.: Stochastic relaxation, Gibbs distribution and the Bayesian restoration of images. IEEE Trans. Pattern Anal. Machine Intell. 6(6):721–741 (1984)

    Article  MATH  Google Scholar 

  7. Hartley, A., Zisserman, A.: Multiple view geometry in computer vision (2 ed.). Cambridge University Press, Cambridge (2006)

  8. Hermans, C., Vanaken, C., Mertens, T., Reeth, F.V., Bekaert, P.: Augmented panoramic video. Comput. Graph. Forum 27(2):281–290 (2008)

    Article  Google Scholar 

  9. Hernandez-Lopez, F.J.: AVScreen: a real-time video augmentation method. Website (2013). http://www.cimat.mx/fcoj23/AVScreenProject/AVSCREEN.html

  10. Hernandez-Lopez, F.J., Rivera, M.: Binary segmentation of video sequences in real time. In: MICAI, pp. 163–168. IEEE Proceedings (2010)

  11. Hernandez-Lopez, F.J., Rivera, M.: Change detection by probabilistic segmentation from monocular view. Mach. Vision Appl. (2012) (to appear)

  12. Juan, O., Keriven, R.: Trimap segmentation for fast and user-friendly alpha matting. In: VLSM, pp. 186–197. Springer-Verlag (2005)

  13. Kolmogorov, V., Criminisi, A., Blake, A., Cross, G., Rother, C.: Bilayer segmentation of binocular stereo video. In: Proceedings of CVPR, pp. 1186–1193 (2005)

  14. Kolmogorov, V., Criminisi, A., Blake, A., Cross, G., Rother, C.: Probabilistic fusion of stereo with color and contrast for bi-layer segmentation. IEEE Trans. Pattern Anal. Machine Intell. 28(9):1480–1492 (2006)

    Article  Google Scholar 

  15. Lee, S., Yun, ID., Lee, S.U.: Robust bilayer video segmentation by adaptive propagation of global shape and local appearance. J. Visual Commun. Image Represent. 21(7):665–676 (2010)

    Article  Google Scholar 

  16. Li, S.: Markov random field modeling in image analysis. Springer-Verlag, Tokyo (2001)

  17. Lucas, B., Kanade, T.: An iterative image registration technique with an application to stereo vision. In: Proceedings of IJCAI, pp. 121–130 (1981)

  18. Marroquin, J.L., Velasco, F.A., Rivera, M., Nakamura, M.: Gauss-markov measure field models for low-level vision. IEEE Trans. Pattern Anal. Mach. Intell. 23(4):337–348 (2001)

    Article  Google Scholar 

  19. Marzotto, R., Fusiello, A., Murino, V.: High resolution video mosaicing with global alignment. In: CVPR (1), pp. 692–698 (2004)

  20. Microsoft: Microsoft research. Website (2011). http://research.microsoft.com/vision/cambridge/i2i/DSWeb.htm

  21. Nocedal, J., Wright, S.J.: Numerical optimization, series in operations research and financial engineering (2006)

  22. NVIDIA: Cuda zone. Website (2012). http://www.nvidia.com/object/cuda_get.html

  23. Prim, R.C.: Shortest connection networks and some generalizations. Bell Syst. Tech. J. 36(6):1389–1401 (1957)

    Google Scholar 

  24. Rivera, M., Dalmau, O.: Variational viewpoint of the quadratic markov measure field models: theory and algorithms. IEEE Trans. Image Process. 21(3):1246–1257 (2012)

    Article  MathSciNet  Google Scholar 

  25. Rivera, M., Dalmau, O., Tago, J.: Image segmentation by convex quadratic programming. In: Proceedings of ICPR, pp. 1–5 (2008)

  26. Rother, C., Kolmogorov, V., Blake, A., "grabcut": interactive foreground extraction using iterated graph cuts. ACM Trans. Graph. 23(3):309–314 (2004)

    Article  Google Scholar 

  27. Rublee, E., Rabaud, V., Konolige, K., Bradski, G.R.: Orb: an efficient alternative to sift or surf. In: ICCV, pp. 2564–2571. IEEE (2011)

  28. Shi, J., Tomasi, C.: Good features to track. In: 1994 IEEE Conference on Computer Vision and Pattern Recognition (CVPR’94), pp. 593–600 (1994)

  29. Sun, J., Zhang, W., Tang, X., Shum, H.Y.: Background cut. In: Proceedings of ECCV, pp. 628–641 (2006)

  30. Szeliski, R., Zabih, R., Scharstein, D., Veksler, O., Kolmogorov, V., Agarwala, A., Tappen, M., Rother, C.A.: comparative study of energy minimization methods for markov random fields with smoothness-based priors. IEEE Trans. Pattern Anal. Mach. Intell. 30(6):1068–1080 (2008)

    Article  Google Scholar 

  31. Tang, Z., Miao, Z., Wan, Y., Jesse, F.F.: Foreground prediction for bilayer segmentation of videos. Pattern Recogn. Lett. 32(14):1720–1734 (2011)

    Article  Google Scholar 

  32. Willowgarage: OpenCV. Website (2012). http://opencv.willowgarage.com/

  33. Yin, P., Criminisi, A., Winn, J., Essa, I.: Bilayer segmentation of webcam videos using tree-based classifiers. IEEE Trans. Pattern Anal. Mach. Intell. 33(1):30–42 (2011)

    Article  Google Scholar 

Download references

Acknowledgments

This work was supported in part by the CONACYT, Mexico (DSc. Scholarship to F.H. and grant 131369-Y to M.R.). The author thanks to the anonymous reviewers for their comments to improve the quality of the paper.

Author information

Authors and Affiliations

  1. Centro de Investigacion en Matematicas AC,  Guanajuato,  Mexico

    Francisco J. Hernandez–Lopez & Mariano Rivera

Authors
  1. Francisco J. Hernandez–Lopez
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mariano Rivera
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Francisco J. Hernandez–Lopez.

Electronic supplementary material

Below is the link to the electronic supplementary material.

DOCX (22 KB)

Rights and permissions

Reprints and permissions

About this article

Cite this article

Hernandez–Lopez, F.J., Rivera, M. AVScreen: a real-time video augmentation method. J Real-Time Image Proc 10, 453–465 (2015). https://doi.org/10.1007/s11554-013-0375-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11554-013-0375-9

Keywords

Search

Navigation