Explorable Representation of Interaction in VR/AR Environments

  • Jakub FlotyńskiEmail author
  • Adrian Nowak
  • Krzysztof Walczak
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10851)


Synthetic 3D content, which is the main element of virtual and augmented reality applications, typically includes objects, which react to interaction with users and other objects. Interaction may result in changes of the objects’ geometry, structure and appearance. Representation of 3D content covering temporal properties can be useful in application domains that benefit from analysis of users’ and objects’ behavior, such as education, training, e-commerce, marketing and merchandising. However, the available approaches do not enable exploration of 3D content with regards to its time-dependent properties. The main contribution of this paper is temporal representation of interaction. The representation is based on semantic web standards and ontologies, which enable use of general as well as domain knowledge for exploration of content. The representation is discussed in the context of an immersive virtual car showroom implemented using headsets and motion tracking devices.


Semantic web Ontologies Headsets Motion tracking 


  1. 1.
    3D Modeling Ontology (2018).
  2. 2.
  3. 3.
    Oculus Rift (2018).
  4. 4.
    OWL 2 web ontology language profiles (second edition). Accessed 07 Mar 2018
  5. 5.
    OpenGL. Accessed 20 Mar 2015
  6. 6.
    Albertoni, R., et al.: Ontology-based searching framework for digital shapes. In: Meersman, R., Tari, Z., Herrero, P. (eds.) OTM 2005. LNCS, vol. 3762, pp. 896–905. Springer, Heidelberg (2005). Scholar
  7. 7.
    Albrecht, S., Wiemann, T., Günther, M., Hertzberg, J.: Matching CAD object models in semantic mapping. In: Proceedings ICRA 2011 Workshop: Semantic Perception, Mapping and Exploration, SPME (2011)Google Scholar
  8. 8.
    Artale, A., Franconi, E.: A survey of temporal extensions of description logics. Ann. Math. Artif. Intell. 30(1–4), 171–210 (2001). Scholar
  9. 9.
    Autodesk: 3ds Max (2017).
  10. 10.
    Away3D: Away3D (2017).
  11. 11.
    Batsakis, S., Petrakis, E., Tachmazidis, I., Antoniou, G.: Temporal representation and reasoning in OWL 2. Semant. Web 8, 1–20 (2009). Scholar
  12. 12.
    Berners-Lee, T., Hendler, J., Lassila, O., et al.: The semantic web. Sci. Am. 284(5), 28–37 (2001)CrossRefGoogle Scholar
  13. 13.
    Catalano, C.E., Mortara, M., Spagnuolo, M., Falcidieno, B.: Semantics and 3D media: current issues and perspectives. Comput. Graph. 35(4), 869–877 (2011)CrossRefGoogle Scholar
  14. 14.
    Chaudhuri, S., Kalogerakis, E., Giguere, S., Funkhouser, T.: Attribit: content creation with semantic attributes. In: Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology, UIST 2013, pp. 193–202. ACM, New York (2013)Google Scholar
  15. 15.
    Ciptawilangga, Y.: Online merchandising and ecommerce with virtual reality simulation of an actual retail location, US Patent App. 12/474, 202, 2 December 2010Google Scholar
  16. 16.
    De Floriani, L., Hui, A., Papaleo, L., Huang, M., Hendler, J.: A semantic web environment for digital shapes understanding. In: Falcidieno, B., Spagnuolo, M., Avrithis, Y., Kompatsiaris, I., Buitelaar, P. (eds.) SAMT 2007. LNCS, vol. 4816, pp. 226–239. Springer, Heidelberg (2007). Scholar
  17. 17.
    De Troyer, O., Kleinermann, F., Pellens, B., Bille, W.: Conceptual modeling for virtual reality. In: Grundy, J., Hartmann, S., Laender, A.H.F., Maciaszek, L., Roddick, J.F. (eds.) Tutorials, Posters, Panels and Industrial Contributions at the 26th International Conference on Conceptual Modeling - ER 2007, CRPIT, Auckland, New Zealand, vol. 83, pp. 3–18. ACS (2007)Google Scholar
  18. 18.
    DFKI, Computer Graphics Lab of the Saarland University, I.V.C.I.: XML3D (2017).
  19. 19.
    Falcidieno, B., Spagnuolo, M., Alliez, P., Quak, E., Vavalis, E., Houstis, C.: Towards the semantics of digital shapes: the AIM@SHAPE approach. In: EWIMT (2004)Google Scholar
  20. 20.
    Flotyński, J.: Semantic modelling of interactive 3D content with domain-specific ontologies. Procedia Comput. Sci. 35, 531–540 (2014). 18th International Conference on Knowledge-Based and Intelligent Information and Engineering SystemsCrossRefGoogle Scholar
  21. 21.
    Flotyński, J., Krzyszkowski, M., Walczak, K.: Semantic composition of 3D content behavior for explorable virtual reality applications. In: Barbic, J., D’Cruz, M., Latoschik, M.E., Slater, M., Bourdot, P. (eds.) EuroVR 2017. LNCS, vol. 10700, pp. 3–23. Springer, Cham (2017). Scholar
  22. 22.
    Flotyński, J., Walczak, K.: Semantic multi-layered design of interactive 3D presentations. In: Proceedings of the Federated Conference on Computer Science and Information Systems, Kraków, Poland, 8–11 September 2013, pp. 541–548. IEEE (2013)Google Scholar
  23. 23.
    Flotyński, J., Walczak, K.: Conceptual knowledge-based modeling of interactive 3D content. Vis. Comput. 31, 1–20 (2014)Google Scholar
  24. 24.
    Flotyński, J., Walczak, K.: Ontology-based creation of 3D content in a service-oriented environment. In: Abramowicz, W. (ed.) BIS 2015. LNBIP, vol. 208, pp. 77–89. Springer, Cham (2015). Scholar
  25. 25.
    Flotyński, J., Walczak, K.: Customization of 3D content with semantic meta-scenes. Graph. Model. 88, 23–39 (2016)MathSciNetCrossRefGoogle Scholar
  26. 26.
    Flotyński, J., Walczak, K.: Knowledge-based representation of 3D content behavior in a service-oriented virtual environment. In: Proceedings of the 22nd International Conference on Web3D Technology, Brisbane, Australia, 5–7 June 2017, Article no. 14. ACM, New York (2017)Google Scholar
  27. 27.
    Flotyński, J., Walczak, K.: Ontology-based representation and modelling of synthetic 3D content: a state-of-the-art review. In: Computer Graphics Forum, pp. 1–25 (2017)CrossRefGoogle Scholar
  28. 28.
    Foundation, B.: Blender (2017).
  29. 29.
    Gutierrez, C., Hurtado, C., Vaisman, A.: Temporal RDF. In: Gómez-Pérez, A., Euzenat, J. (eds.) ESWC 2005. LNCS, vol. 3532, pp. 93–107. Springer, Heidelberg (2005). Scholar
  30. 30.
    Gutiérrez, M., García-Rojas, A., Thalmann, D., Vexo, F., Moccozet, L., Magnenat-Thalmann, N., Mortara, M., Spagnuolo, M.: An ontology of virtual humans: Incorporating semantics into human shapes. Vis. Comput. 23(3), 207–218 (2007)CrossRefGoogle Scholar
  31. 31.
    Hu, X., Liu, X., He, Z., Zhang, J.: Batch modeling of 3D city based on Esri cityengine. In: Smart and Sustainable City 2013, ICSSC 2013, IET International Conference on IET (2013)Google Scholar
  32. 32.
    Jiang, H., Xu, W., Mao, T., Li, C., Xia, S., Wang, Z.: A semantic environment model for crowd simulation in multilayered complex environment. In: Proceedings of the 16th ACM Symposium on Virtual Reality Software and Technology, pp. 191–198. ACM (2009)Google Scholar
  33. 33.
    Kalogerakis, E., Christodoulakis, S., Moumoutzis, N.: Coupling ontologies with graphics content for knowledge driven visualization. In: VR 2006 Proceedings of the IEEE conference on Virtual Reality, Alexandria, Virginia, USA, 25–29 March 2006, pp. 43–50 (2006)Google Scholar
  34. 34.
    Klein, M., Fensel, D.: Ontology versioning on the semantic web. In: Proceedings of the First International Conference on Semantic Web Working, SWWS 2001, pp. 75–91., Aachen, Germany (2001).
  35. 35.
    Kowalski, R., Sergot, M.: A Logic-based calculus of events. In: Schmidt, J.W., Thanos, C. (eds.) Foundations of Knowledge Base Management, Topics in Information Systems, pp. 23–55. Springer, Heidelberg (1989). Scholar
  36. 36.
    Kowalski, R.A., Sadri, F.: The situation calculus and event calculus compared. In: ILPS, vol. 94, pp. 539–553 (1994)Google Scholar
  37. 37.
    Latoschik, M.E., Blach, R.: Semantic modelling for virtual worlds - a novel paradigm for realtime interactive systems? In: Proceedings of the ACM VRST 2008, pp. 17–20 (2008)Google Scholar
  38. 38.
    Lugrin, J.L., Cavazza, M.: Making sense of virtual environments: action representation, grounding and common sense. In: Proceedings of the 12th International Conference on Intelligent User Interfaces, IUI 2007, pp. 225–234. ACM, New York (2007)Google Scholar
  39. 39.
    Lui, T.W., Piccoli, G., Ives, B.: Marketing strategies in virtual worlds. SIGMIS Database 38(4), 77–80 (2007). Scholar
  40. 40.
    Microsoft: Direct3D 11.1 features (2017).
  41. 41.
    Noy, N., Rector, A., Hayes, P., Welty, C.: Defining n-ary relations on the semantic web. Accessed 06 Mar 2018
  42. 42.
    Oracle: Java3D (2017).
  43. 43.
    Otto, K.: Semantic virtual environments. In: Special Interest Tracks and Posters of the 14th International Conference on World Wide Web, Chiba, Japan, 10–14 May 2005, pp. 1036–1037 (2005)Google Scholar
  44. 44.
    Pellens, B., De Troyer, O., Kleinermann, F.: CoDePA: a conceptual design pattern approach to model behavior for X3D worlds. In: Proceedings of the 13th International Symposium on 3D web technology, Los Angeles, 09–10 August 2008, pp. 91–99 (2008)Google Scholar
  45. 45.
    Pellens, B., Kleinermann, F., De Troyer, O.: A development environment using behavior patterns to facilitate building 3D/VR applications. In: Proceedings of the 6th Australasian Conference on Interactive Entertainment, IE 2009, pp. 8:1–8:8. ACM (2009)Google Scholar
  46. 46.
    Pittarello, F., De Faveri, A.: Semantic description of 3D environments: a proposal based on web standards. In: Proceedings of the Eleventh International Conference on 3D Web Technology, Web3D 2006, pp. 85–95. ACM, New York (2006)Google Scholar
  47. 47.
    Rabattu, P.Y., Massé, B., Ulliana, F., Rousset, M.C., Rohmer, D., Léon, J.C., Palombi, O.: My Corporis Fabrica Embryo: an ontology-based 3D spatio-temporal modeling of human embryo development. J. Biomed. Semant. 6(1), 36 (2015). Scholar
  48. 48.
    Rumiński, D.: An experimental study of spatial sound usefulness in searching and navigating through ar environments. Virtual Real. 19, 223–233 (2015). Scholar
  49. 49.
    Rumiński, D., Walczak, K.: Semantic contextual augmented reality environments. In: The 13th IEEE International Symposium on Mixed and Augmented Reality (ISMAR 2014), ISMAR 2014, pp. 401–404. IEEE (2014)Google Scholar
  50. 50.
    Shanahan, M.: The event calculus explained. In: Wooldridge, M.J., Veloso, M. (eds.) Artificial Intelligence Today. LNCS (LNAI), vol. 1600, pp. 409–430. Springer, Heidelberg (1999). Scholar
  51. 51.
    Sikos, L.F.: 3D model indexing in videos for content-based retrieval via X3D-based semantic enrichment and automated reasoning. In: Proceedings of the 22nd International Conference on 3D Web Technology, Web3D 2017, pp. 19:1–19:7. ACM, New York (2017).
  52. 52.
    Spagnuolo, M., Falcidieno, B.: The role of ontologies for 3D media applications. In: Kompatsiaris, Y., Hobson, P. (eds.) Semantic Multimedia and Ontologies. Springer, London (2008). Scholar
  53. 53.
    Spagnuolo, M., Falcidieno, B.: 3D media and the semantic web. IEEE Intell. Syst. 24(2), 90–96 (2009)CrossRefGoogle Scholar
  54. 54.
    Technologies, U.: Unity (2017).
  55. 55.
    Trellet, M., Ferey, N., Baaden, M., Bourdot, P.: Interactive visual analytics of molecular data in immersive environments via a semantic definition of the content and the context. In: 2016 Workshop on Immersive Analytics (IA), pp. 48–53. IEEE (2016)Google Scholar
  56. 56.
    Tutenel, T., Bidarra, R., Smelik, R.M., De Kraker, K.J.: The role of semantics in games and simulations. Comput. Entertain. (CIE) 6(4), 57 (2008)Google Scholar
  57. 57.
    Van Gool, L., Leibe, B., Müller, P., Vergauwen, M., Weise, T.: 3D challenges and a non-in-depth overview of recent progress. In: 3DIM, pp. 118–132 (2007)Google Scholar
  58. 58.
  59. 59.
  60. 60.
  61. 61.
    W3C: OWL. Accessed 24 Mar 2015
  62. 62.
  63. 63.
    W3C: RDFS. Accessed 24 Mar 2015
  64. 64.
    Walczak, K., Rumiński, D., Flotyński, J.: Building contextual augmented reality environments with semantics. In: Proceedings of the 20th International Conference on Virtual Systems and Multimedia, Hong Kong, 9–12 September 2014 (2014)Google Scholar
  65. 65.
    Welty, C., Fikes, R.: A reusable ontology for fluents in OWL. In: Proceedings of the 2006 Conference on Formal Ontology in Information Systems: Proceedings of the Fourth International Conference (FOIS 2006), pp. 226–236. IOS Press, Amsterdam (2006).
  66. 66.
    Wiebusch, D., Latoschik, M.E.: Enhanced decoupling of components in intelligent realtime interactive systems using ontologies. In: Software Engineering and Architectures for Realtime Interactive Systems (SEARIS), Proceedings of the IEEE Virtual Reality 2012 Workshop (2012)Google Scholar
  67. 67.
    Zaid, L.A., Kleinermann, F., De Troyer, O.: Applying semantic web technology to feature modeling. In: Proceedings of the 2009 ACM Symposium on Applied Computing, SAC 2009, pp. 1252–1256. ACM (2009)Google Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • Jakub Flotyński
    • 1
    Email author
  • Adrian Nowak
    • 1
  • Krzysztof Walczak
    • 1
  1. 1.Poznań University of Economics and BusinessPoznańPoland

Personalised recommendations