Skip to main content
SpringerLink
Log in

Real-scene-constrained virtual scene layout synthesis for mixed reality

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Given a real source scene and a virtual target scene, the real-scene-constrained virtual scene layout synthesis problem is defined as how to re-synthesize the layout of the virtual furniture in the virtual scene to form a new virtual scene such that the new scene not only looks similar to the input real and virtual scenes but also is interactive. The goal of this problem is to maximize interactivity and fidelity which are contradictory. To solve this problem, we propose a real-scene-constrained virtual scene layout synthesis method to synthesize the layout of the virtual furniture in the new virtual scene. We split the scene layout synthesis process into 3 interrelated steps: scene matching, matched furniture layout generating, and unmatched furniture layout generating. For scene matching, we propose a deep scene matching network to predict the matching relationship between real and virtual furniture. For matched furniture layout generating, we propose a layout parameters optimization algorithm to predict suitable layouts of the matched virtual furniture. For unmatched furniture layout generating, we propose a deep scene generating network to predict suitable layouts of unmatched virtual furniture. We evaluate the quality of our method to synthesize scenes of different kinds and sizes. The results show that, compared with the heuristic rules-based method, our method has better matching accuracy and location accuracy. We also design a user study to evaluate the interactivity and fidelity. Compared to the manual method and the heuristic rules-based method, our method has a significant improvement in interactivity and fidelity.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Algorithm 1
Algorithm 2
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Data availability

The datasets generated during and/or analyzed during the current study are available from the corresponding author on reasonable request.

References

  1. Huang, C.-K., Chen, Y.-L., Shen, I.-C., Chen, B.-Y.: Retargeting 3d objects and scenes with a general framework. Comput. Graph. Forum 35(7), 33–42 (2016)

    Article  Google Scholar 

  2. Lin, J., Cohen-Or, D., Zhang, H., Liang, C., Sharf, A., Deussen, O., Chen, B.: Structure-preserving retargeting of irregular 3d architecture. ACM Trans. Graph. 30(6), 1–10 (2011)

    Article  Google Scholar 

  3. Dong, Z.-C., Wu, W., Xu, Z., Sun, Q., Yuan, G., Liu, L., Fu, X.-M.: Tailored reality: perception-aware scene restructuring for adaptive vr navigation. ACM Trans. Graph. 40(5), 1–15 (2021)

    Article  Google Scholar 

  4. Dong, Z.-C., Fu, X.-M., Zhang, C., Wu, K., Liu, L.: Smooth assembled mappings for large-scale real walking. ACM Trans. Graph. 36(6), 1–13 (2017)

    Article  Google Scholar 

  5. Wang, K., Lin, Y.-A., Weissmann, B., Savva, M., Chang, A.X., Ritchie, D.: Planit: planning and instantiating indoor scenes with relation graph and spatial prior networks. ACM Trans. Graph. 38(4), 1–15 (2019)

    Article  Google Scholar 

  6. Fu, Q., Chen, X., Wang, X., Wen, S., Zhou, B., Fu, H.: Adaptive synthesis of indoor scenes via activity-associated object relation graphs. ACM Trans. Graph. 36(6), 1–13 (2017)

    Google Scholar 

  7. Fisher, M., Savva, M., Li, Y., Hanrahan, P., Nießner, M.: Activity-centric scene synthesis for functional 3d scene modeling. ACM Trans. Graph. 34(6), 1–13 (2015)

    Article  Google Scholar 

  8. Cant, R.J., Langensiepen, C.S.: Methods for automated object placement in virtual scenes. In: 2009 11th International Conference on Computer Modelling and Simulation, pp. 431–436 (2009)

  9. Shamir, A., Sorkine, O.: Visual media retargeting. In: ACM SIGGRAPH ASIA 2009 Courses. SIGGRAPH ASIA ’09. Association for Computing Machinery, New York, NY, USA (2009). doi:10.1145/1665817.1665828

  10. Ma, L., Lin, W., Deng, C., Ngan, K.N.: Image retargeting quality assessment: a study of subjective scores and objective metrics. IEEE J. Sel. Topics Signal Process. 6(6), 626–639 (2012)

    Article  Google Scholar 

  11. Dong, Z.-C., Fu, X.-M., Yang, Z., Liu, L.: Redirected smooth mappings for multiuser real walking in virtual reality. ACM Trans. Graph. 38(5), 1–17 (2019)

    Article  Google Scholar 

  12. Merrell, P., Schkufza, E., Li, Z., Agrawala, M., Koltun, V.: Interactive furniture layout using interior design guidelines. ACM Trans. Graph. 30(4), 1–10 (2011)

    Article  Google Scholar 

  13. Peng, C.-H., Yang, Y.-L., Wonka, P.: Computing layouts with deformable templates. ACM Trans. Graph. 33(4), 1–11 (2014)

    Article  MATH  Google Scholar 

  14. Merrell, P., Schkufza, E., Koltun, V.: Computer-generated residential building layouts. ACM Trans. Graph. 29(6) (2010)

  15. Fu, Q., Fu, H., Deng, Z., Li, X.: Indoor layout programming via virtual navigation detectors. Sci. China Inf. Sci. 65(8), 1–2 (2022)

    Article  Google Scholar 

  16. Zhang, S., Han, Z., Lai, Y., Zwicker, M., Zhang, H.: Stylistic scene enhancement GAN: mixed stylistic enhancement generation for 3d indoor scenes. Vis. Comput. 35(6–8), 1157–1169 (2019)

    Article  Google Scholar 

  17. Vasylevska, K., Kaufmann, H.: Towards efficient spatial compression in self-overlapping virtual environments. In: 2017 IEEE Symposium on 3D User Interfaces (3DUI), pp. 12–21 (2017)

  18. Zhao, X., Su, Z., Komura, T., Yang, X.: Building hierarchical structures for 3d scenes with repeated elements. Vis. Comput. 36(2), 361–374 (2020)

    Article  Google Scholar 

  19. Wang, K., Savva, M., Chang, A.X., Ritchie, D.: Deep convolutional priors for indoor scene synthesis. ACM Trans. Graph. 37(4), 1–14 (2018)

    Google Scholar 

  20. Li, M., Patil, A.G., Xu, K., Chaudhuri, S., Khan, O., Shamir, A., Tu, C., Chen, B., Cohen-Or, D., Zhang, H.: Grains: generative recursive autoencoders for indoor scenes. ACM Trans. Graph. 38(2), 1–16 (2019)

    Article  Google Scholar 

  21. Feng, S., Mostafa, H., Nassar, M., Majumdar, S., Tripathi, S.: Exploiting long-term dependencies for generating dynamic scene graphs. In: 2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 5119–5128 (2023)

  22. Zanfir, A., Sminchisescu, C.: Deep learning of graph matching. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2684–2693 (2018)

  23. Yan, J., Yin, X.-C., Lin, W., Deng, C., Zha, H., Yang, X.: A short survey of recent advances in graph matching. In: Proceedings of the 2016 ACM on International Conference on Multimedia Retrieval. ICMR ’16, pp. 167–174. Association for Computing Machinery, New York, NY, USA (2016)

  24. Loiola, E.M., de Abreu, N.M.M., Boaventura-Netto, P.O., Hahn, P., Querido, T.: A survey for the quadratic assignment problem. Eur. J. Oper. Res. 176(2), 657–690 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  25. Cho, M., Lee, J., Lee, K.M.: Reweighted random walks for graph matching. In: Proceedings of the 11th European Conference on Computer Vision: Part V. ECCV’10, pp. 492–505. Springer, Berlin, Heidelberg (2010)

  26. Leordeanu, M., Hebert, M.: A spectral technique for correspondence problems using pairwise constraints. In: Tenth IEEE International Conference on Computer Vision (ICCV’05) Volume 1, vol. 2, pp. 1482–14892 (2005)

  27. Hahn, P., Grant, T., Hall, N.: A branch-and-bound algorithm for the quadratic assignment problem based on the Hungarian method. Eur. J. Oper. Res. 108(3), 629–640 (1998)

    Article  MATH  Google Scholar 

  28. Wang, T., Ling, H., Lang, C., Feng, S.: Graph matching with adaptive and branching path following. IEEE Trans. Pattern Anal. Mach. Intell. 40(12), 2853–2867 (2018)

    Article  Google Scholar 

  29. Kushinsky, Y., Maron, H., Dym, N., Lipman, Y.: Sinkhorn algorithm for lifted assignment problems. SIAM J. Imag. Sci. 12(2), 716–735 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  30. Liu, C., Niu, D., Yang, X., Zhao, X.: Graph matching based on feature and spatial location information. Vis. Comput. 39(2), 711–722 (2023)

    Article  Google Scholar 

  31. Li, C., Tang, Y., Zou, X., Zhang, P., Lin, J., Lian, G., Pan, Y.: A novel agricultural machinery intelligent design system based on integrating image processing and knowledge reasoning. Appl. Sci. 12(15), 7900 (2022)

    Article  Google Scholar 

  32. Ji, Z., Chen, K., He, Y., Pang, Y., Li, X.: Heterogeneous memory enhanced graph reasoning network for cross-modal retrieval. Sci. China Inf. Sci. 65(7), 1–13 (2022)

    Article  MathSciNet  Google Scholar 

  33. Wu, T., Duan, F., Chang, L., Lu, K.: Human-object interaction detection via interactive visual-semantic graph learning. Sci. China Inf. Sci. 65(6), 1–2 (2022)

    Article  Google Scholar 

  34. Zhou, D., Liu, Y., Li, X., Zhang, C.: Single-image super-resolution based on local biquadratic spline with edge constraints and adaptive optimization in transform domain. Vis. Comput. 38(1), 119–134 (2022)

    Article  Google Scholar 

  35. Chen, Y., Zhang, Q., Guan, Z., Zhao, Y., Chen, W.: Gemvis: a visual analysis method for the comparison and refinement of graph embedding models. Vis. Comput. 38(9), 3449–3462 (2022)

    Article  Google Scholar 

  36. Wang, R., Yan, J., Yang, X.: Learning combinatorial embedding networks for deep graph matching. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 3056–3065 (2019)

  37. Wu, Z., Pan, S., Long, G., Jiang, J., Zhang, C.: Graph wavenet for deep spatial-temporal graph modeling. In: Proceedings of the 28th International Joint Conference on Artificial Intelligence. IJCAI’19, pp. 1907–1913. AAAI Press, Palo Alto, CA (2019)

  38. Cao, S., Lu, W., Xu, Q.: Deep neural networks for learning graph representations. In: Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence. AAAI’16, pp. 1145–1152. AAAI Press, Palo Alto, CA (2016)

  39. Chiang, W.-L., Liu, X., Si, S., Li, Y., Bengio, S., Hsieh, C.-J.: Cluster-gcn: An efficient algorithm for training deep and large graph convolutional networks. In: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery Amp; Data Mining. KDD ’19, pp. 257–266. Association for Computing Machinery, New York, NY, USA (2019)

  40. Li, Q., Han, Z., Wu, X.-M.: Deeper insights into graph convolutional networks for semi-supervised learning. In: Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence. AAAI’18/IAAI’18/EAAI’18. AAAI Press, Palo Alto, CA (2018)

  41. Zhang, M., Cui, Z., Neumann, M., Chen, Y.: An end-to-end deep learning architecture for graph classification. In: Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence. AAAI’18/IAAI’18/EAAI’18. AAAI Press, Palo Alto, CA (2018)

  42. Huang, W., Zhang, T., Rong, Y., Huang, J.: Adaptive sampling towards fast graph representation learning. In: Proceedings of the 32nd International Conference on Neural Information Processing Systems. NIPS’18, pp. 4563–4572. Curran Associates Inc., Red Hook, NY, USA (2018)

  43. Chen, J., Ma, T., Xiao, C.: FastGCN: Fast learning with graph convolutional networks via importance sampling. In: International Conference on Learning Representations (2018)

  44. Gilmer, J., Schoenholz, S.S., Riley, P.F., Vinyals, O., Dahl, G.E.: Neural message passing for quantum chemistry. In: Proceedings of the 34th International Conference on Machine Learning - Volume 70. ICML’17, pp. 1263–1272. JMLR.org, New York, NY (2017)

  45. Spinelli, I., Scardapane, S., Uncini, A.: Adaptive propagation graph convolutional network. IEEE Trans. Neural Netw. Learn. Syst. 32(10), 4755–4760 (2021)

    Article  Google Scholar 

  46. Scarselli, F., Gori, M., Tsoi, A.C., Hagenbuchner, M., Monfardini, G.: The graph neural network model. IEEE Trans. Neural Netw. 20(1), 61–80 (2009)

    Article  Google Scholar 

  47. Wei, H., Meng, L.: An accurate stereo matching method based on color segments and edges. Pattern Recognit. 133, 108996 (2023)

    Article  Google Scholar 

  48. Ryan Prescott Adams, R.S.Z.: Ranking via sinkhorn propagation (2011)

  49. Newell, A., Yang, K., Deng, J.: Stacked hourglass networks for human pose estimation. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) Computer Vision-ECCV 2016, pp. 483–499. Springer, Cham (2016)

    Chapter  Google Scholar 

  50. Fu, H., Jia, R., Gao, L., Gong, M., Zhao, B., Maybank, S., Tao, D.: 3d-future: 3d furniture shape with texture. Int. J. Comput. Vis. 129, 3313–3337 (2021)

    Article  Google Scholar 

Download references

Funding

This work is supported by the National Natural Science Foundation of China through Project 61932003, 62372026, by Beijing Science and Technology Plan Project Z221100007722004, and by National Key R &D plan 2019YFC1521102.

Author information

Authors and Affiliations

  1. State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, XueYuan Road, Beijing, 100191, Beijing, China

    Runze Fan, Lili Wang & Xinda Liu

  2. Peng Cheng Laboratory, Xingke Road, Shenzhen, 518055, Shenzhen, China

    Lili Wang

  3. Macao Polytechnic University, R. de Luís Gonzaga Gomes, Macao, 999078, Macao, China

    Sio Kei Im & Chan Tong Lam

Authors
  1. Runze Fan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lili Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xinda Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Sio Kei Im
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Chan Tong Lam
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lili Wang.

Ethics declarations

Conflict of interest

The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses or interpretation of data; in the writing of the manuscript; or in the decision to publish the results.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file 1 (mp4 56872 KB)

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Fan, R., Wang, L., Liu, X. et al. Real-scene-constrained virtual scene layout synthesis for mixed reality. Vis Comput (2023). https://doi.org/10.1007/s00371-023-03167-4

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s00371-023-03167-4

Keywords

Search

Navigation