Skip to main content
SpringerLink
Log in

Fast Intra-Frame Video Splicing for Occlusion Removal in Diminished Reality

  • Conference paper
  • First Online:
Virtual Reality and Mixed Reality (EuroXR 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13484))

Included in the following conference series:

Abstract

In a real world scene objects of interest might be occluded by other objects. Diminished reality (DR) aims to remove such occluders. A popular approach is to acquire the geometry of the occluded scene, and then to render it from the user’s viewpoint, effectively erasing the occluder. However, the approach is ill-suited for scenes with intricate and dynamic geometry, which cannot be acquired quickly, completely, and with only minimal equipment. This paper proposes a method to erase an occluder in a primary video by splicing in pixels from a secondary video. For each frame, the method finds the region in the secondary frame that corresponds to the occluder shadow, and integrates it seamlessly into the primary frame. Precise matching of the occluder contour is achieved by a novel pipeline with a tracking, global alignment, and local alignment stages. The result is a continuous multiperspective frame, which shows most of the scene from the primary viewpoint, except for the part hidden by the occluder, which is shown from the secondary viewpoint. A high quality multiperspective transparency effect is achieved for complex scenes, without the high cost of 3D acquisition. When compared with other DR methods, the proposed method shows fewer artifacts and better continuity.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Adobe photoshop. https://www.adobe.com/products/photoshop.html. Accessed 30 Apr 2020

  2. Álvarez, H., Arrieta, J., Oyarzun, D.: Towards a diminished reality system that preserves structures and works in real-time. In: VISIGRAPP (4: VISAPP), pp. 334–343 (2017)

    Google Scholar 

  3. Andre, E., Hlavacs, H.: Diminished reality based on 3D-scanning. In: van der Spek, E., Göbel, S., Do, E.Y.-L., Clua, E., Baalsrud Hauge, J. (eds.) ICEC-JCSG 2019. LNCS, vol. 11863, pp. 3–14. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-34644-7_1

    Chapter  Google Scholar 

  4. Avery, B., Piekarski, W., Thomas, B.H.: Visualizing occluded physical objects in unfamiliar outdoor augmented reality environments. In: Proceedings of the 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality, pp. 1–2. IEEE Computer Society (2007)

    Google Scholar 

  5. Barnes, C., Shechtman, E., Goldman, D.B., Finkelstein, A.: The generalized PatchMatch correspondence algorithm. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010. LNCS, vol. 6313, pp. 29–43. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-15558-1_3

    Chapter  Google Scholar 

  6. Barnum, P., Sheikh, Y., Datta, A., Kanade, T.: Dynamic seethroughs: synthesizing hidden views of moving objects. In: 2009 8th IEEE International Symposium on Mixed and Augmented Reality, pp. 111–114. IEEE (2009)

    Google Scholar 

  7. Bay, H., Tuytelaars, T., Van Gool, L.: SURF: speeded up robust features. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3951, pp. 404–417. Springer, Heidelberg (2006). https://doi.org/10.1007/11744023_32

    Chapter  Google Scholar 

  8. Bugeau, A., Bertalmío, M., Caselles, V., Sapiro, G.: A comprehensive framework for image inpainting. IEEE Trans. Image Process. 19(10), 2634–2645 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  9. Criminisi, A., Perez, P., Toyama, K.: Object removal by exemplar-based inpainting. In: 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings, vol. 2, pp. II–II. IEEE (2003)

    Google Scholar 

  10. Enomoto, A., Saito, H.: Diminished reality using multiple handheld cameras. In: Proceedings of ACCV, vol. 7, pp. 130–135 (2007)

    Google Scholar 

  11. Fischler, M.A., Bolles, R.C.: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 24(6), 381–395 (1981)

    Article  MathSciNet  Google Scholar 

  12. Hasegawa, K., Saito, H.: Diminished reality for hiding a pedestrian using hand-held camera. In: 2015 IEEE International Symposium on Mixed and Augmented Reality Workshops, pp. 47–52. IEEE (2015)

    Google Scholar 

  13. Herling, J., Broll, W.: High-quality real-time video inpaintingwith pixmix. IEEE Trans. Visual Comput. Graphics 20(6), 866–879 (2014)

    Article  Google Scholar 

  14. Holland, P.W., Welsch, R.E.: Robust regression using iteratively reweighted least-squares. Commun. Stat. Theor. Methods 6(9), 813–827 (1977)

    Article  MATH  Google Scholar 

  15. Iizuka, S., Simo-Serra, E., Ishikawa, H.: Globally and locally consistent image completion. ACM Trans. Graph. (ToG) 36(4), 1–14 (2017)

    Article  Google Scholar 

  16. Jampour, M., Li, C., Yu, L.F., Zhou, K., Lin, S., Bischof, H.: Face inpainting based on high-level facial attributes. Comput. Vis. Image Underst. 161, 29–41 (2017)

    Article  Google Scholar 

  17. Kameda, Y., Takemasa, T., Ohta, Y.: Outdoor see-through vision utilizing surveillance cameras. In: Proceedings of the 3rd IEEE/ACM International Symposium on Mixed and Augmented Reality, pp. 151–160. IEEE Computer Society (2004)

    Google Scholar 

  18. Li, Z., Zhu, H., Cao, L., Jiao, L., Zhong, Y., Ma, A.: Face inpainting via nested generative adversarial networks. IEEE Access 7, 155462–155471 (2019)

    Article  Google Scholar 

  19. Li, Z., Wang, Y., Guo, J., Cheong, L.F., Zhou, S.Z.: Diminished reality using appearance and 3d geometry of internet photo collections. In: 2013 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 11–19. IEEE (2013)

    Google Scholar 

  20. Liu, G., Reda, F.A., Shih, K.J., Wang, T.C., Tao, A., Catanzaro, B.: Image inpainting for irregular holes using partial convolutions. In: The European Conference on Computer Vision (ECCV) (2018)

    Google Scholar 

  21. Meerits, S., Saito, H.: Real-time diminished reality for dynamic scenes. In: 2015 IEEE International Symposium on Mixed and Augmented Reality Workshops, pp. 53–59. IEEE (2015)

    Google Scholar 

  22. Mei, C., Sommerlade, E., Sibley, G., Newman, P., Reid, I.: Hidden view synthesis using real-time visual slam for simplifying video surveillance analysis. In: 2011 IEEE International Conference on Robotics and Automation, pp. 4240–4245. IEEE (2011)

    Google Scholar 

  23. Mori, S., Ikeda, S., Saito, H.: A survey of diminished reality: techniques for visually concealing, eliminating, and seeing through real objects. IPSJ Trans. Comput. Vis. Appl. 9(1), 1–14 (2017). https://doi.org/10.1186/s41074-017-0028-1

    Article  Google Scholar 

  24. Mori, S., Maezawa, M., Ienaga, N., Saito, H.: Diminished hand: a diminished reality-based work area visualization. In: 2017 IEEE Virtual Reality (VR), pp. 443–444. IEEE (2017)

    Google Scholar 

  25. Mori, S., Shibata, F., Kimura, A., Tamura, H.: Efficient use of textured 3d model for pre-observation-based diminished reality. In: 2015 IEEE International Symposium on Mixed and Augmented Reality Workshops, pp. 32–39. IEEE (2015)

    Google Scholar 

  26. Muja, M., Lowe, D.G.: Fast approximate nearest neighbors with automatic algorithm configuration. VISAPP (1) 2(331–340), 2 (2009)

    Google Scholar 

  27. Nakajima, Y., Mori, S., Saito, H.: Semantic object selection and detection for diminished reality based on slam with viewpoint class. In: 2017 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct), pp. 338–343. IEEE (2017)

    Google Scholar 

  28. Nocedal, J., Wright, S.: Numerical Optimization. Springer, New York (2006). https://doi.org/10.1007/978-0-387-40065-5

  29. Queguiner, G., Fradet, M., Rouhani, M.: Towards mobile diminished reality. In: 2018 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), pp. 226–231. IEEE (2018)

    Google Scholar 

  30. Rameau, F., Ha, H., Joo, K., Choi, J., Park, K., Kweon, I.S.: A real-time augmented reality system to see-through cars. IEEE Trans. Visual Comput. Graphics 22(11), 2395–2404 (2016)

    Article  Google Scholar 

  31. Singhal, A., et al.: Modern information retrieval: a brief overview. IEEE Data Eng. Bull. 24(4), 35–43 (2001)

    Google Scholar 

  32. Takemura, M., Ohta, Y.: Diminishing head-mounted display for shared mixed reality. In: Proceedings of the 1st International Symposium on Mixed and Augmented Reality, p. 149. IEEE Computer Society (2002)

    Google Scholar 

  33. Tiefenbacher, P., Sirch, M., Rigoll, G.: Mono camera multi-view diminished reality. In: 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1–8. IEEE (2016)

    Google Scholar 

  34. Wang, L., Jin, H., Yang, R., Gong, M.: Stereoscopic inpainting: joint color and depth completion from stereo images. In: 2008 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8. IEEE (2008)

    Google Scholar 

  35. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)

    Article  Google Scholar 

  36. Wexler, Y., Shechtman, E., Irani, M.: Space-time video completion. In: Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition 2004, CVPR 2004, vol. 1, pp. I-I. IEEE (2004)

    Google Scholar 

  37. Wu, M.L., Popescu, V.: Rgbd temporal resampling for real-time occlusion removal. In: Proceedings of the ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games, p. 7. ACM (2019)

    Google Scholar 

  38. Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., Huang, T.S.: Generative image inpainting with contextual attention. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5505–5514 (2018)

    Google Scholar 

  39. Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., Huang, T.S.: Free-form image inpainting with gated convolution. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 4471–4480 (2019)

    Google Scholar 

  40. Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: CVPR (2018)

    Google Scholar 

  41. Zhang, Z.: A flexible new technique for camera calibration. IEEE Trans. Pattern Anal. Mach. Intell. 22(11), 1330–1334 (2000)

    Article  Google Scholar 

  42. Zokai, S., Esteve, J., Genc, Y., Navab, N.: Multiview paraperspective projection model for diminished reality. In: Proceedings of the 2nd IEEE/ACM International Symposium on Mixed and Augmented Reality, p. 217. IEEE Computer Society (2003)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Purdue University, West Lafayette, IN, 47906, USA

    Chengyuan Lin & Voicu Popescu

Authors
  1. Chengyuan Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Voicu Popescu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Chengyuan Lin .

Editor information

Editors and Affiliations

  1. University of Bremen, Bremen, Germany

    Gabriel Zachmann

  2. Universitat Politècnica de València, Valencia, Spain

    Mariano Alcañiz Raya

  3. University of Paris-Saclay, Orsay Cedex, France

    Patrick Bourdot

  4. INSA, IRISA, University of Rennes, Rennes Cedex, France

    Maud Marchal

  5. University of Utah, Salt Lake City, UT, USA

    Jeanine Stefanucci

  6. Shanghai Jiao Tong University, Shanghai, China

    Xubo Yang

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Lin, C., Popescu, V. (2022). Fast Intra-Frame Video Splicing for Occlusion Removal in Diminished Reality. In: Zachmann, G., Alcañiz Raya, M., Bourdot, P., Marchal, M., Stefanucci, J., Yang, X. (eds) Virtual Reality and Mixed Reality. EuroXR 2022. Lecture Notes in Computer Science, vol 13484. Springer, Cham. https://doi.org/10.1007/978-3-031-16234-3_7

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-16234-3_7

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-16233-6

  • Online ISBN: 978-3-031-16234-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation