Semantic Composition of 3D Content Behavior for Explorable Virtual Reality Applications

  • Jakub FlotyńskiEmail author
  • Marcin Krzyszkowski
  • Krzysztof Walczak
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10700)


Virtual reality (VR) applications become increasingly popular in various application domains because of the possibilities of realistic immersive presentation and interaction with virtual objects as well as the diversity of advanced, relatively cheap devices. The semantic web, which is an important trend in the current web development, requires effective exploration of content distributed across different applications. Exploration of 3D content, which is the main part of VR applications, is a complex task, as the geometry, structure and appearance of the content may interactively evolve over time. Although a number of solutions are available for implementation of VR applications, the approaches have not been intended for on-demand exploration of behavior-rich 3D content in real time. In this paper, we present the development pipeline of explorable VR applications, which is based on semantic composition of 3D content activities into more complex behavior. The resulting applications are based on Prolog, which is a well-established knowledge representation language, and they can be queried for time-dependent 3D content features using domain-specific concepts. The approach has been implemented using the OpenStage 2 motion capture system and the Unity game engine. It can be used in different application domains and improve the integration of VR with the semantic web.


3D content exploration Semantic web Behavior modeling Event calculus Motion capture Unity 


  1. 1.
  2. 2.
    Albertoni, R., Papaleo, L., Pitikakis, M., Robbiano, F., Spagnuolo, M., Vasilakis, G.: Ontology-based searching framework for digital shapes. In: Meersman, R., Tari, Z., Herrero, P. (eds.) OTM 2005. LNCS, vol. 3762, pp. 896–905. Springer, Heidelberg (2005). CrossRefGoogle Scholar
  3. 3.
    Albrecht, S., Wiemann, T., Günther, M., Hertzberg, J.: Matching CAD object models in semantic mapping. In: Proceedings ICRA 2011 Workshop: Semantic Perception, Mapping and Exploration, SPME (2011)Google Scholar
  4. 4.
    Attene, M., Robbiano, F., Spagnuolo, M., Falcidieno, B.: Semantic annotation of 3D surface meshes based on feature characterization. In: Falcidieno, B., Spagnuolo, M., Avrithis, Y., Kompatsiaris, I., Buitelaar, P. (eds.) SAMT 2007. LNCS, vol. 4816, pp. 126–139. Springer, Heidelberg (2007). CrossRefGoogle Scholar
  5. 5.
    Autodesk: 3ds Max (2017).
  6. 6.
    Autodesk: Motion Builder (2017).
  7. 7.
    Away3D: Away3d (2017).
  8. 8.
    Bille, W., Pellens, B., Kleinermann, F., De Troyer, O.: Intelligent modelling of virtual worlds using domain ontologies. In: Proceedings of the Workshop of Intelligent Computing (WIC), Held in Conjunction with the MICAI 2004 Conference, Mexico City, Mexico, pp. 272–279 (2004)Google Scholar
  9. 9.
    Bille, W.: Conceptual modeling of complex objects for virtual environments. Ph.D. thesis, Vrije Universiteit Brussel (2006–2007)Google Scholar
  10. 10.
    Bille, W., De Troyer, O., Kleinermann, F., Pellens, B., Romero, R.: Using ontologies to build virtual worlds for the web. In: Isaías, P.T., Karmakar, N., Rodrigues, L., Barbosa, P. (eds.) ICWI, IADIS, pp. 683–690 (2004)Google Scholar
  11. 11.
    Chu, Y., Li, T.: Using pluggable procedures and ontology to realize semantic virtual environments 2.0. In: Proceedings of the 7th ACM SIGGRAPH International Conference on Virtual-Reality Continuum and Its Applications in Industry, VRCAI 2008, pp. 27:1–27:6. ACM, New York (2008)Google Scholar
  12. 12.
    Chu, Y., Li, T.: Realizing semantic virtual environments with ontology and pluggable procedures. In: Applications of Virtual Reality (2012)Google Scholar
  13. 13.
    De Floriani, L., Hui, A., Papaleo, L., Huang, M., Hendler, J.: A semantic web environment for digital shapes understanding. In: Falcidieno, B., Spagnuolo, M., Avrithis, Y., Kompatsiaris, I., Buitelaar, P. (eds.) SAMT 2007. LNCS, vol. 4816, pp. 226–239. Springer, Heidelberg (2007). CrossRefGoogle Scholar
  14. 14.
    De Troyer, O., Bille, W., Romero, R., Stuer, P.: On generating virtual worlds from domain ontologies. In: Proceedings of the 9th International Conference on Multi-media Modeling, Taipei, Taiwan, pp. 279–294 (2003)Google Scholar
  15. 15.
    De Troyer, O., Kleinermann, F., Pellens, B., Bille, W.: Conceptual modeling for virtual reality. In: Grundy, J., Hartmann, S., Laender, A.H.F., Maciaszek, L., Roddick, J.F. (eds.) Tutorials, Posters, Panels and Industrial Contributions at the 26th International Conference on Conceptual Modeling - ER 2007. CRPIT, vol. 83, pp. 3–18. ACS, Auckland (2007)Google Scholar
  16. 16.
    DFKI: Computergraphics Lab of the Saarland University, Intel Visual Computing Institute: Xml3d (2017).
  17. 17.
    Falcidieno, B., Spagnuolo, M., Alliez, P., Quak, E., Vavalis, E., Houstis, C.: Towards the semantics of digital shapes: the AIM@SHAPE approach. In: EWIMT (2004)Google Scholar
  18. 18.
    Flotyński, J.: Harvesting of semantic metadata from distributed 3D web content. In: Proceedings of the 6th International Conference on Human System Interaction (HSI), Sopot, Poland, 06–08 June 2013. IEEE (2013)Google Scholar
  19. 19.
    Flotyński, J.: Semantic modelling of interactive 3D content with domain-specific ontologies. Procedia Comput. Sci. 35, 531–540 (2014). 18th International Conference on Knowledge-Based and Intelligent Information and Engineering SystemsCrossRefGoogle Scholar
  20. 20.
    Flotyński, J., Walczak, K.: Attribute-based semantic descriptions of interactive 3D web content. In: Kiełtyka, L. (ed.) Information Technologies in Organizations - Management and Applications of Multimedia, pp. 111–138. Wydawnictwa Towarzystwa Naukowego Organizacji i Kierownictwa - Dom Organizatora (2013)Google Scholar
  21. 21.
    Flotyński, J., Walczak, K.: Microformat and microdata schemas for interactive 3D web content. In: Ganzha, M., Maciaszek, L., Paprzycki, M. (eds.) Proceedings of the 2013 FedCSIS Kraków, Poland, 8–11 September 2013, vol. 1, pp. 549–556. Polskie Towarzystwo Informatyczne (2013)Google Scholar
  22. 22.
    Flotyński, J., Walczak, K.: Conceptual knowledge-based modeling of interactive 3D content. Vis. Comput. 31(10), 1287–1306 (2015). SpringerCrossRefGoogle Scholar
  23. 23.
    Flotyński, J., Walczak, K.: Semantic representation of multi-platform 3D content. Comput. Sci. Inf. Syst. 11(4), 1555–1580 (2014)CrossRefGoogle Scholar
  24. 24.
    Flotyński, J., Walczak, K.: Ontology-based creation of 3D content in a service-oriented environment. In: Abramowicz, W. (ed.) BIS 2015. LNBIP, vol. 208, pp. 77–89. Springer, Cham (2015). CrossRefGoogle Scholar
  25. 25.
    Flotyński, J., Walczak, K.: Customization of 3D content with semantic meta-scenes. Graph. Models 88, 23–39 (2016)MathSciNetCrossRefGoogle Scholar
  26. 26.
    Flotyński, J., Walczak, K.: Knowledge-based representation of 3D content behavior in a service-oriented virtual environment. In: Proceedings of the 22nd International Conference on Web 3D Technology, Brisbane, Australia, 5–7 June 2017. ACM, New York (2017). Article No. 14Google Scholar
  27. 27.
    Flotyński, J., Walczak, K.: Ontology-based representation and modelling of synthetic 3D content: a state-of-the-art review. In: Computer Graphics Forum, pp. 1–25 (2017)Google Scholar
  28. 28.
    Foundation, B.: Blender (2017).
  29. 29.
    García-Rojas, A., Vexo, F., Thalmann, D., Raouzaiou, A., Karpouzis, K., Kollias, S.: Emotional body expression parameters in virtual human ontology. In: Proceedings of 1st International Workshop on Shapes and Semantics, Matsushima, Japan, June 2006, pp. 63–70 (2006)Google Scholar
  30. 30.
    Gutiérrez, M.: Semantic virtual environments. EPFL (2005)Google Scholar
  31. 31.
    Gutiérrez, M., García-Rojas, A., Thalmann, D., Vexo, F., Moccozet, L., Magnenat-Thalmann, N., Mortara, M., Spagnuolo, M.: An ontology of virtual humans: incorporating semantics into human shapes. Vis. Comput. 23(3), 207–218 (2007)CrossRefGoogle Scholar
  32. 32.
    Kalogerakis, E., Christodoulakis, S., Moumoutzis, N.: Coupling ontologies with graphics content for knowledge driven visualization. In: Proceedings of the IEEE Conference on Virtual Reality, VR 2006, Alexandria, Virginia, USA, 25–29 March 2006, pp. 43–50 (2006)Google Scholar
  33. 33.
    Kapahnke, P., Liedtke, P., Nesbigall, S., Warwas, S., Klusch, M.: ISReal: an open platform for semantic-based 3D simulations in the 3D internet. In: Patel-Schneider, P.F., Pan, Y., Hitzler, P., Mika, P., Zhang, L., Pan, J.Z., Horrocks, I., Glimm, B. (eds.) ISWC 2010. LNCS, vol. 6497, pp. 161–176. Springer, Heidelberg (2010). CrossRefGoogle Scholar
  34. 34.
    Kowalski, R., Sergot, M.: A logic-based calculus of events. In: Schmidt, J.W., Thanos, C. (eds.) Foundations of Knowledge Base Management Topics in Information Systems, pp. 23–55. Springer, Heidelberg (1989). Google Scholar
  35. 35.
    Latoschik, M.E., Tramberend, H.: Simulator X: a scalable and concurrent software platform for intelligent realtime interactive systems. In: Proceedings of the IEEE VR 2011 (2011)Google Scholar
  36. 36.
    Microsoft: Direct3d 11.1 features (2017).
  37. 37.
    Oracle: Java3d (2017).
  38. 38.
    Otto, K.: Semantic virtual environments. In: Special Interest Tracks and Posters of the 14th International Conference on World Wide Web, Chiba, Japan, 10–14 May 2005, pp. 1036–1037 (2005)Google Scholar
  39. 39.
    Otto, K.: The semantics of multi-user virtual environments. In: Proceedings of the Workshop towards Semantic Virtual Environments (2005)Google Scholar
  40. 40.
    Papaleo, L., De Floriani, L., Hendler, J., Hui, A.: Towards a semantic web system for understanding real world representations. In: Proceedings of the Tenth International Conference on Computer Graphics and Artificial Intelligence (2007)Google Scholar
  41. 41.
    Pellens, B., De Troyer, O., Bille, W., Kleinermann, F., Romero, R.: An ontology-driven approach for modeling behavior in virtual environments. In: Meersman, R., Tari, Z., Herrero, P. (eds.) OTM 2005. LNCS, vol. 3762, pp. 1215–1224. Springer, Heidelberg (2005). CrossRefGoogle Scholar
  42. 42.
    Pittarello, F., De Faveri, A.: Semantic description of 3D environments: a proposal based on web standards. In: Proceedings of the Eleventh International Conference on 3D Web Technology, Web3D 2006, pp. 85–95. ACM, New York (2006)Google Scholar
  43. 43.
    Rabattu, P.Y., Massé, B., Ulliana, F., Rousset, M.C., Rohmer, D., Léon, J.C., Palombi, O.: My Corporis Fabrica Embryo: An ontology-based 3D spatio-temporal modeling of human embryo development. J. Biomed. Semant. 6(1), 36 (2015). BioMed CentralCrossRefGoogle Scholar
  44. 44.
    Reitmayr, G., Schmalstieg, D.: Semantic world models for ubiquitous augmented reality. In: Proceedings of Workshop Towards Semantic Virtual Environments’ (SVE) 2005 (2005)Google Scholar
  45. 45.
    Robbiano, F., Attene, M., Spagnuolo, M., Falcidieno, B.: Part-based annotation of virtual 3D shapes. In: 2013 International Conference on Cyberworlds, pp. 427–436 (2007)Google Scholar
  46. 46.
    Rumiński, D.: An experimental study of spatial sound usefulness in searching and navigating through AR environments. Virtual Real. 19, 223–233 (2015). CrossRefGoogle Scholar
  47. 47.
    Rumiński, D., Walczak, K.: Semantic contextual augmented reality environments. In: The 13th IEEE International Symposium on Mixed and Augmented Reality, ISMAR 2014, pp. 401–404. IEEE (2014)Google Scholar
  48. 48.
    Rumiński, D., Walczak, K.: Semantic model for distributed augmented reality services. In: Proceedings of the 22th International Conference on 3D Web Technology, Proceeding Web3D 2017, Brisbane, Australia, 5–7 June 2017. ACM, New York (2017, accepted for publication)Google Scholar
  49. 49.
    Russell, S., Norvig, P., Intelligence, A.: A Modern Approach. Pearson, London (2009)Google Scholar
  50. 50.
    Shanahan, M.: The event calculus explained. In: Wooldridge, M.J., Veloso, M. (eds.) Artificial Intelligence Today. LNCS (LNAI), vol. 1600, pp. 409–430. Springer, Heidelberg (1999). CrossRefGoogle Scholar
  51. 51.
    Spagnuolo, M., Falcidieno, B.: The role of ontologies for 3D media applications. In: Kompatsiaris, Y., Hobson, P. (eds.) Semantic Multimedia and Ontologies. Springer, London (2008). Google Scholar
  52. 52.
    Trimble: Sketchup (2017).
  53. 53.
    Vasilakis, G., García-Rojas, A., Papaleo, L., Catalano, C.E., Robbiano, F., Spagnuolo, M., Vavalis, M., Pitikakis, M.: Knowledge-based representation of 3D media. Int. J. Softw. Eng. Knowl. Eng. 20(5), 739–760 (2010)CrossRefGoogle Scholar
  54. 54.
  55. 55.
  56. 56.
    Walczak, K., Flotyński, J.: On-demand generation of 3D content based on semantic meta-scenes. In: De Paolis, L.T., Mongelli, A. (eds.) AVR 2014. LNCS, vol. 8853, pp. 313–332. Springer, Cham (2014). Google Scholar
  57. 57.
    Walczak, K., Flotyński, J.: Semantic query-based generation of customized 3D scenes. In: Proceedings of the 20th International Conference on 3D Web Technology, Web3D 2015, pp. 123–131. ACM, New York (2015).
  58. 58.
    Walczak, K., Rumiński, D., Flotyński, J.: Building contextual augmented reality environments with semantics. In: Proocedings of the 20th International Conference on Virtual Systems and Multimedia, Hong Kong, 9–12 September 2014 (2014)Google Scholar
  59. 59.
    Whissel, C.M.: Emotion: theory, research and experience. In: The Dictionary of Affect in Language, vol. 4. New York (1989)Google Scholar
  60. 60.
    Wiebusch, D., Latoschik, M.E.: Enhanced decoupling of components in intelligent realtime interactive systems using ontologies. In: Proceedings of the IEEE Virtual Reality 2012 Workshop on Software Engineering and Architectures for Realtime Interactive Systems (SEARIS) (2012)Google Scholar
  61. 61.
    Zaid, L.A., Kleinermann, F., De Troyer, O.: Applying semantic web technology to feature modeling. In: Proceedings of the 2009 ACM Symposium on Applied Computing, SAC 2009, pp. 1252–1256. ACM (2009)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Jakub Flotyński
    • 1
    Email author
  • Marcin Krzyszkowski
    • 1
  • Krzysztof Walczak
    • 1
  1. 1.Poznań University of Economics and BusinessPoznańPoland

Personalised recommendations