The Visual Computer

, Volume 33, Issue 2, pp 193–208 | Cite as

Diminished reality for augmented reality interior design

  • Sanni Siltanen
Original Article


A modular real-time diminished reality pipeline for indoor applications is presented. The pipeline includes a novel inpainting method which requires no prior information of the textures behind the object to be diminished. The inpainting method operates on rectified images and adapts to scene illumination. In typically challenging illumination situations, the method produces more realistic results in indoor scenes than previous approaches. Modularity enables using alternative implementations in different stages and adapting the pipeline for different applications. Finally, practical solutions to problems occurring in diminished reality applications, for example interior design, are discussed.


Diminished reality Inpainting Augmented reality  Interior design AR 



Dr. Timo Tossavainen provided valuable comments during the writing of this article.

Supplementary material

371_2015_1174_MOESM1_ESM.mpg (41.1 mb)
Supplementary material 1 (mpg 42052 KB)
371_2015_1174_MOESM2_ESM.mpg (902 kb)
Supplementary material 2 (mpg 902 KB)


  1. 1.
    Alvar: A library for virtual and augmented reality.
  2. 2.
    Bao, S.Y., Furlan, A., Fei-Fei, L., Savarese, S.: Understanding the 3d layout of a cluttered room from multiple images. In: 2014 IEEE Winter Conference on Applications of Computer Vision (WACV), Steamboat Springs, CO, 24–26 March 2014, pp. 690–697 (2014)Google Scholar
  3. 3.
    Bhatewara, N., Kumar, P., Agrawal, A.: Intelligent video inpainting system for texture reconstruction. In: 2013 4th International Conference on Computer and Communication Technology (ICCCT), Allahabad, 20–22 Sept 2013, pp. 60–65 (2013)Google Scholar
  4. 4.
    Criminisi, A., Perez, P., Toyama, K.: Object removal by exemplar-based inpainting. In: 2013 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), 18–20 June 2003, vol. 2, pp. II-721–II-728 (2003)Google Scholar
  5. 5.
    Dou, M., Fuchs, H.: Temporally enhanced 3d capture of room-sized dynamic scenes with commodity depth cameras. In: Virtual Reality (VR), IEEE, Minneapolis, MN, USA, 29 March-2 April 2014, pp. 39–44 (2014)Google Scholar
  6. 6.
    Efros, A.A., Leung, T.K.: Texture synthesis by non-parametric sampling. In: Proceedings of the International Conference on Computer Vision, vol. 2. ICCV ’99, p. 1033. IEEE Computer Society, Washington, DC (1999)Google Scholar
  7. 7.
    Hartley, R.I., Zisserman, A.: Multiple View Geometry in Computer Vision. Cambridge University Press, Cambridge (2000)zbMATHGoogle Scholar
  8. 8.
    Henry, P., Krainin, M., Herbst, E., Ren, X., Fox, D.: Rgb-d mapping: using depth cameras for dense 3d modeling of indoor environments. In: Khatib, O., Kumar, V., Sukhatme, G. (eds.) Experimental Robotics, Springer Tracts in Advanced Robotics, vol. 79, pp. 477–491. Springer, Berlin (2014)Google Scholar
  9. 9.
    Herling, J., Broll, W.: Advanced self-contained object removal for realizing real-time diminished reality in unconstrained environments. In: 2014 9th IEEE International Symposium on Mixed and Augmented Reality (ISMAR) Seoul, South Korea, 13–16 Oct 2010, pp. 207–212 (2010)Google Scholar
  10. 10.
    Herling, J., Broll, W.: Pixmix: a real-time approach to high-quality diminished reality. In: 2012 IEEE International Symposium on Mixed and Augmented Reality (ISMAR) Atlanta, GA, USA, 5–8 Nov 2012, pp. 141–150 (2012)Google Scholar
  11. 11.
    Herling, J., Broll, W.: High-quality real-time video inpainting with pixmix. IEEE Trans. Vis. Comput. Graph. 20(6), 866–879 (2014). ID: 1Google Scholar
  12. 12.
    Jarusirisawad, S., Hosokawa, T., Yokoya, N.: Diminished reality using plane-sweep algorithm with weakly-calibrated cameras. Prog. Inform. 7, 11–20 (2010)CrossRefGoogle Scholar
  13. 13.
    Kawai, N., Sato, T., Yokoya, N.: From image inpainting to diminished reality. In: Shumaker, R., Lackey, S. (eds.) Virtual, Augmented and Mixed Reality. Designing and Developing Virtual and Augmented Environments. Lecture Notes in Computer Science, vol. 8525, pp. 363–374. Springer International Publishing, Berlin (2014)Google Scholar
  14. 14.
    Kawai, N., Yamasaki, M., Sato, T., Yokoya, N.: Ar marker hiding based on image inpainting and reflection of illumination changes. In: 2012 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 293–294 (2012)Google Scholar
  15. 15.
    Kholgade, N., Simon, T., Efros, A., Sheikh, Y.: 3d object manipulation in a single photograph using stock 3d models. ACM Trans. Graph. 33(4), 127:1–127:12 (2014)Google Scholar
  16. 16.
    Korkalo, O., Aittala, M., Siltanen, S.: Light-weight marker hiding for augmented reality. In: 2010 9th IEEE International Symposium on Mixed and Augmented Reality (ISMAR) Seoul, South Korea, 13–16 Oct 2010, pp. 247–248 (2010)Google Scholar
  17. 17.
    Li, Z., Wang, Y., Guo, J., Cheong, L.F., Zhou, S.: Diminished reality using appearance and 3d geometry of internet photo collections. In: 2013 IEEE International Symposium on Mixed and Augmented Reality (ISMAR) Adelaide, SA, 1–4 Oct 2013, pp. 11–19 (2013) Google Scholar
  18. 18.
    Liu, H., Philipose, M., Sun, M.T.: Automatic objects segmentation with rgb-d cameras. J. Vis. Commun. Image Represent. 25(4), 709–718 (2014)Google Scholar
  19. 19.
    Lu, B.V., Kakuta, T., Kawakami, R., Oishi, T., Ikeuchi, K.: Foreground and shadow occlusion handling for outdoor augmented reality. In: 2010 9th IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Seoul, South Korea, 13–16 Oct 2010 pp. 109–118 (2010)Google Scholar
  20. 20.
    Mann, S.: Mediated reality with implementations for everyday life.PRESENCE: Teleoperators and Virtual Environments, MIT Press (2002)Google Scholar
  21. 21.
    Mura, C., Mattausch, O., Villanueva, A.J., Gobbetti, E., Pajarola, R.: Automatic room detection and reconstruction in cluttered indoor environments with complex room layouts. Comput. Graph. 44, 20–32 (2014)CrossRefGoogle Scholar
  22. 22.
    Pintore, G., Gobbetti, E.: Effective mobile mapping of multi-room indoor structures. Vis. Comput. 30(6–8), 707–716 (2014)CrossRefGoogle Scholar
  23. 23.
    Shepard, D.: A two-dimensional interpolation function for irregularly-spaced data. In: Proceedings of the 1968 23rd ACM National Conference. ACM ’68, pp. 517–524. ACM, New York (1968)Google Scholar
  24. 24.
    Siltanen, S.: Texture generation over the marker area. In: IEEE/ACM International Symposium on Mixed and Augmented Reality, 2006. ISMAR 2006, Santa Barbara, CA, 22–25 Oct 2006, pp. 253–254 (2006)Google Scholar
  25. 25.
    Siltanen, S., Oksman, V., Ainasoja, M.: User-centered design of augmented reality interior design service. Int. J. Arts Sci. 6(1), 547–563 (2013)Google Scholar
  26. 26.
    Siltanen, S., Saraspaa, H., Karvonen, J.: [Demo] a complete interior design solution with diminished reality. In: 2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Munich, Germany, 10–12 Sept 2014, pp. 371–372 (2014)Google Scholar
  27. 27.
    Simon, G., Berger, M.O.: Interactive building and augmentation of piecewise planar environments using the intersection lines. Vis. Comput. 27(9), 827–841 (2011)CrossRefGoogle Scholar
  28. 28.
    Yang, M.D., Chao, C.F., Huang, K.S., Lu, L.Y., Chen, Y.P.: Image-based 3d scene reconstruction and exploration in augmented reality. Autom. Constr. 33, 48–60 (2013). Augmented Reality in Architecture, Engineering, and ConstructionGoogle Scholar
  29. 29.
    Zou, C., Peng, X., Lv, H., Chen, S., Fu, H., Liu, J.: Sketch-based 3-d modeling for piecewise planar objects in single images. Comput. Graph. 46, 130–137 (2015)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2015

Authors and Affiliations

  1. 1.VTT Technical Research Centre of Finland LtdVTTFinland

Personalised recommendations