Skip to main content
SpringerLink
Log in

Semi-Automatic 2D to 3D Video Conversion

  • Chapter
  • First Online:
Smart Algorithms for Multimedia and Imaging

Part of the book series: Signals and Communication Technology ((SCT))

  • 389 Accesses

Abstract

In this chapter, the process of converting conventional video sequences to stereo format is considered. Key frame detection, depth assignment, depth propagation, motion vector estimation, background inpainting, and stereo synthesis comprise the stereo conversion pipeline. We also consider several aspects of content production, such as stereo shooting and stereo conversion. The depth propagation algorithm is based on patch matching using patch hash codes, and the results are compared with motion vector-based depth propagation results. Motion vector estimation is an essential algorithm that is used to reveal video clip attributes, and it plays an important role during conversion. We propose an optical flow algorithm based on a primal-dual optimization algorithm. This work is based on the authors’ experience during a large-scale project at the Samsung Research centre in support of newly appearing TVs on the market, which were equipped with a “3D-ready” feature.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 119.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 159.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 159.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  • Appia, V., Batur, U.: Fully automatic 2D to 3D conversion with aid of high-level image features. In: Stereoscopic Displays and Applications XXV, vol. 9011, p. 90110W (2014)

    Chapter  Google Scholar 

  • Baker, S., Scharstein, D., Lewis, J.P., Roth, S., Black, M.J., Szeliski, R.: A database and evaluation methodology for optical flow. Int. J. Comp. Vision. 92(1), 1–31 (2011)

    Article  Google Scholar 

  • Bertalmio, M., Sapiro, G., Caselles, V., Ballester, C.: Image inpainting. In: Proceedings of the 27th Annual Conference on Computer graphics and Interactive Techniques, p. 417 (2000)

    Google Scholar 

  • Bugeau, A., Piracés, P.G.I., d'Hondt, O., Hervieu, A., Papadakis, N., Caselles, V.: Coherent background video inpainting through Kalman smoothing along trajectories. In: Proceedings of 2010–15th International Workshop on Vision, Modeling, and Visualization, p. 123 (2010)

    Google Scholar 

  • Butler D.J., Wulff J., Stanley G.B., Black M.J.: A Naturalistic Open Source Movie for Optical Flow Evaluation. In: Fitzgibbon A., Lazebnik S., Perona P., Sato Y., Schmid C. (eds) Computer Vision – ECCV 2012. ECCV 2012. Lecture Notes in Computer Science, vol 7577. Springer, Berlin, Heidelberg (2012)

    Google Scholar 

  • Cao, X., Li, Z., Dai, Q.: Semi-automatic 2D-to-3D conversion using disparity propagation. IEEE Trans. Broadcast. 57, 491–499 (2011)

    Article  Google Scholar 

  • Chambolle, A., Pock, T.: A first-order primal-dual algorithm for convex problems with applications to imaging. J. Math. Imag. Vision. 40(1), 120–145 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  • Criminisi, A., Pérez, P., Toyama, K.: Region filling and object removal by exemplar-based image inpainting. IEEE Trans. Image Process. 13(9), 1200–1212 (2004)

    Article  Google Scholar 

  • Feng, J., Ma, H., Hu, J., Cao, L., Zhang, H.: Superpixel based depth propagation for semi-automatic 2D-to-3D video conversion. In: Proceedings of IEEE Third International Conference on Networking and Distributed Computing, pp. 157–160 (2012)

    Google Scholar 

  • Feng, Z., Chao, Z., Huamin, Y., Yuying, D.: Research on fully automatic 2D to 3D method based on deep learning. In: Proceedings of the IEEE 2nd International Conference on Automation, Electronics and Electrical Engineering, pp. 538–541 (2019)

    Google Scholar 

  • Fischler, M.A., Bolles, R.C.: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM. 24(6), 381–395 (1981)

    Article  MathSciNet  Google Scholar 

  • Harman, P.V., Flack, J., Fox, S., Dowley, M.: Rapid 2D-to-3D conversion. In: Stereoscopic displays and virtual reality systems IX International Society for Optics and Photonics, vol. 4660, pp. 78–86 (2002)

    Chapter  Google Scholar 

  • Ignatov, A., Bucha, V., Rychagov, M.: Disparity estimation in real-time 3D acquisition and reproduction system. In: Proceedings of the International Conference on Computer Graphics «Graphicon 2009», pp. 61–68 (2009)

    Google Scholar 

  • Irony, R., Cohen-Or, D., Lischinski, D.: Colorization by example. In: Proceedings of the Sixteenth Eurographics conference on Rendering Techniques, pp. 201–210 (2005)

    Google Scholar 

  • Korman, S., Avidan, S.: Coherency sensitive hashing. IEEE Trans. Pattern Anal. Mach. Intell. 38(6), 1099–1112 (2015)

    Article  Google Scholar 

  • Muelle, M., Zill, F., Kauff, P.: Adaptive cross-trilateral depth map filtering. In: Proceedings of the IEEE 3DTV Conference: The True Vision-Capture, Transmission and Display of 3D Video, pp. 1–4 (2010)

    Google Scholar 

  • Pérez, P., Gangnet, M., Blake, A.: Poisson image editing. In: ACM SIGGRAPH Papers, pp. 313–318 (2003)

    Google Scholar 

  • Pohl, P., Molchanov, A., Shamsuarov, A., Bucha, V.: Spatio-temporal video background inpainting. Electron. Imaging. 15, 1–5 (2016)

    Article  Google Scholar 

  • Pohl, P., Sirotenko, M., Tolstaya, E., Bucha, V.: Edge preserving motion estimation with occlusions correction for assisted 2D to 3D conversion. In: Image Processing: Algorithms and Systems XII, 9019, pp. 901–906 (2014)

    Google Scholar 

  • Shiratori, T., Matsushita, Y., Tang, X., Kang, S.: Video completion by motion field transfer. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 1, p. 411 (2006)

    Google Scholar 

  • Sun, J., Xie, J., Li, J., Liu, W.: A key-frame selection method for semi-automatic 2D-to-3D vonversion. In: Zhang, W., Yang, X., Xu, Z., An, P., Liu, Q., Lu, Y. (eds.) Advances on Digital Television and Wireless Multimedia Communications. Communications in Computer and Information Science, vol. 331. Springer, Berlin, Heidelberg (2012)

    Google Scholar 

  • Sun, D., Roth, S., Black, M.J.: Secrets of optical flow estimation and their principles. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 2432–2439 (2010a)

    Google Scholar 

  • Sun, D., Sudderth, E., Black, M.: Layered image motion with explicit occlusions, temporal consistency, and depth ordering. In: Proceedings of the 24th Annual Conference on Neural Information Processing Systems, pp. 2226–2234 (2010b)

    Google Scholar 

  • Teed, Z., Deng, J.: Raft: Recurrent all-pairs field transforms for optical flow. In: Proceedings of the European Conference on Computer Vision, pp. 402–419 (2020)

    Google Scholar 

  • Telea, A.: An image inpainting technique based on the fast marching method. J. Graph. Tools. 9(1) (2004)

    Google Scholar 

  • Tolstaya E.: Implementation of Coherency Sensitive Hashing algorithm. (2020). Accessed on 03 October 2020. https://github.com/ktolstaya/PyCSH

  • Tolstaya, E., Hahn, S.-H.: Method and system for selecting key frames from video sequences. RU Patent 2,493,602 (in Russian) (2012)

    Google Scholar 

  • Tolstaya, E., Pohl, P., Rychagov, M.: Depth propagation for semi-automatic 2d to 3d conversion. In: Proceedings of SPIE Three-Dimensional Image Processing, Measurement, and Applications, vol. 9393, p. 939303 (2015)

    Google Scholar 

  • Varekamp, C., Barenbrug, B.: Improved depth propagation for 2D to 3D video conversion using key-frames. In: Proceedings of the 4th European Conference on Visual Media Production (2007)

    Google Scholar 

  • Vatolin, D.: Why Does 3D Lead to the Headache? / Part 8: Defocus and Future of 3D (in Russian) (2019). Accessed on 03 October 2020. https://habr.com/ru/post/472782/

  • Vatolin, D., Bokov, A., Erofeev, M., Napadovsky, V.: Trends in S3D-movie quality evaluated on 105 films using 10 metrics. Electron. Imaging. 2016(5), 1–10 (2016)

    Article  Google Scholar 

  • Vatolin, D.: Why Does 3D Lead to the Headache? / Part 2: Discomfort because of Video Quality (in Russian) (2015a). Accessed on 03 October 2020. https://habr.com/en/post/377709/

  • Vatolin, D.: Why Does 3D Lead to the Headache? / Part 4: Parallax (in Russian) (2015b). Accessed on 03 October 2020. https://habr.com/en/post/378387/

  • Voronov, A., Vatolin, D., Sumin, D., Napadovsky, V., Borisov, A.: Methodology for stereoscopic motion-picture quality assessment. In: Proceedings of SPIE Stereoscopic Displays and Applications XXIV, vol. 8648, p. 864810 (2013)

    Google Scholar 

  • Wang, D., Liu, J., Sun, J., Liu, W., Li, Y.: A novel key-frame extraction method for semi-automatic 2D-to-3D video conversion. In: Proceedings of the IEEE international Symposium on Broadband Multimedia Systems and Broadcasting, pp. 1–5 (2012)

    Google Scholar 

  • Werlberger, M., Pock, T., Bischof, H.: Motion estimation with non-local total variation regularization. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 2464–2471 (2010)

    Google Scholar 

  • Wexler, Y., Shechtman, E., Irani, M.: Space-time completion of video. IEEE Trans. Pattern Anal. Mach. Intell. 29(3), 463–476 (2007)

    Article  Google Scholar 

  • Xie, J., Girshick, R., Farhadi, A.: Deep3d: Fully automatic 2d-to-3d video conversion with deep convolutional neural networks. In: Proceedings of the European Conference on Computer Vision, pp. 842–857 (2016)

    Google Scholar 

  • Yuan, H.: Robust semi-automatic 2D-to-3D image conversion via residual-driven optimization. EURASIP J. Image Video Proc. 1, 66 (2018)

    Article  Google Scholar 

  • Zhao, S., Sheng, Y., Dong, Y., Chang, E., Xu, Y.: MaskFlownet: asymmetric feature matching with learnable occlusion mask. In: Proceedings of the CVPR, vol. 1, pp. 6277–6286 (2020)

    Google Scholar 

  • Zitnick, C.L., Kang, S.B., Uyttendaele, M., Winder, S., Szeliski, R.: High-quality video view interpolation using a layered representation. ACM Transactions on Graphics. 23(3) (2004)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Samsung R&D Institute Rus (SRR), Moscow, Russia

    Petr Pohl

  2. Aramco Innovations LLC, Moscow, Russia

    Ekaterina V. Tolstaya

Authors
  1. Petr Pohl
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ekaterina V. Tolstaya
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Petr Pohl .

Editor information

Editors and Affiliations

  1. National Research University of Electronic Technology (MIET), Moscow, Russia

    Michael N. Rychagov

  2. Aramco Innovations LLC, Moscow, Russia

    Ekaterina V. Tolstaya

  3. Google Research, New York, NY, USA

    Mikhail Y. Sirotenko

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Pohl, P., Tolstaya, E.V. (2021). Semi-Automatic 2D to 3D Video Conversion. In: Rychagov, M.N., Tolstaya, E.V., Sirotenko, M.Y. (eds) Smart Algorithms for Multimedia and Imaging. Signals and Communication Technology. Springer, Cham. https://doi.org/10.1007/978-3-030-66741-2_4

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-66741-2_4

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-66740-5

  • Online ISBN: 978-3-030-66741-2

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation