Spatiotemporal Reasoning for Complex Video Event Recognition in Content-Based Video Retrieval

  • Leslie F. Sikos
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 639)


Ontology-based representation of video scenes and events indicates a promising direction in content-based video retrieval. However, the multimedia ontologies described in the literature often lack formal grounding, and none of them are suitable for representing complex video scenes. This issue can be partially addressed using SWRL rules, which, however, can lead to undecidability. This paper presents a hybrid description logic-based architecture that employs general, spatial, temporal, and fuzzy axioms for video scene representation and automated reasoning-based scene interpretation, while achieving a favorable tradeoff between expressivity and reasoning complexity.


Video semantics Spatiotemporal annotation Automated reasoning Video scene interpretation Ontology-based content retrieval Temporal description logic Spatial description logic Fuzzy description logic 


  1. 1.
    Sikos, L.F., Powers, D.M.W.: Knowledge-driven video information retrieval with LOD: from semi-structured to structured video metadata. In: Eighth Workshop on Exploiting Semantic Annotations in Information Retrieval, pp. 35–37. ACM, New York (2015). doi: 10.1145/2810133.2810141
  2. 2.
    Sikos, L.F.: 3D model indexing in videos for content-based retrieval via X3D-based semantic enrichment and automated reasoning. In: 22nd International Conference on 3D Web Technology. ACM, New York (2017). doi: 10.1145/3055624.3075943
  3. 3.
    Hotz, L., Neumann, B., Terzic, K.: High-level expectations for low-level image processing. In: Dengel, A.R., Berns, K., Breuel, T.M., Bomarius, F., Roth-Berghofer, T.R. (eds.) KI 2008: Advances in Artificial Intelligence. LNCS, vol. 5243, pp. 87–94. Springer, Heidelberg (2008). doi: 10.1007/978-3-540-85845-4_11
  4. 4.
    Gómez-Romero, J., Patricio, M.A., García, J., Molina, J.M.: Ontology based context representation and reasoning for object tracking and scene interpretation in video. Expert Syst. Appl. 38(6), 7494–7510 (2011). doi: 10.1016/j.eswa.2010.12.118 CrossRefGoogle Scholar
  5. 5.
    Möller, R., Neumann, B.: Ontology-based reasoning techniques for multimedia interpretation and retrieval. In: Kompatsiaris, Y., Hobson, P. (eds.) Semantic Multimedia and Ontologies: Theory and Application, pp. 55–98. Springer, London (2008). doi: 10.1007/978-1-84800-076-6_3
  6. 6.
    Sikos, L.F.: Mastering Structured Data on the Semantic Web: From HTML5 Microdata to Linked Open Data. Apress, New York (2015). ISBN 978-1-4842-1050-5. doi: 10.1007/978-1-4842-1049-9
  7. 7.
    Sikos, L.F.: A novel approach to multimedia ontology engineering for automated reasoning over audiovisual LOD datasets. In: Nguyễn, N.T., Trawiński, B., Fujita, H., Hong, T.-P. (eds.) Intelligent Information and Database Systems. LNCS, vol. 9621, pp. 3–12. Springer, Heidelberg (2016). doi: 10.1007/978-3-662-49381-6_1
  8. 8.
    Tani, M.Y.K., Lablack, A., Ghomari, A., Bilasco, I.M.: Events detection using a video surveillance ontology and a rule-based approach. In: Agapito, L., Bronstein, M.M., Rother, C. (eds.) Computer Vision – ECCV 2014 Workshops. LNCS, vol. 8926, pp. 299–308, Springer, Cham (2014). doi: 10.1007/978-3-319-16181-5_21
  9. 9.
    Sikos, L.F.: Description logics in multimedia reasoning. Springer, Cham (2017). ISBN 978-3-319-54065-8, doi: 10.1007/978-3-319-54066-5
  10. 10.
    Haarslev, V.: A logic-based formalism for reasoning about visual representations. J. Visual Lang. Comput. 4(10), 421–445 (1999). doi: 10.1006/jvlc.1999.0133 CrossRefGoogle Scholar
  11. 11.
    Kaplunova, A., Haarslev, V., Möller, R.: Adding ternary complex roles to \( \mathcal{A}\mathcal{L}\mathcal{C}_{RP}^{{(\mathcal{D})}}. \) In: 2002 International Workshop on Description Logics, Toulouse, France (2002)Google Scholar
  12. 12.
    Na, K.-S., Kong, H., Cho, M.: Multimedia information retrieval based on spatiotemporal relationships using description logics for the Semantic Web. Int. J. Intell. Syst. 21(7), 679–692 (2006). doi: 10.1002/int.20153 CrossRefMATHGoogle Scholar
  13. 13.
    Wessel, M.: Qualitative spatial reasoning with the \( \mathcal{A}\mathcal{L}\mathcal{C}\mathcal{I}_{{_{RCC} }} \) family—first results and unanswered questions (2003). Accessed 16 Mar 2017
  14. 14.
    Cristani, M., Gabrielli, N.: Practical issues of description logics for spatial reasoning. In: AAAI Spring Symposium, Stanford University, Stanford (2011)Google Scholar
  15. 15.
    Bai, L., Lao, S., Zhang, W., Jones, G.J.F., Smeaton, A.F.: Video semantic content analysis framework based on ontology combined MPEG-7. In: Boujemaa, N., Detyniecki, M., Nürnberger, A. (eds.) Adaptive Multimedia Retrieval: Retrieval, User, and Semantics. LNCS, vol. 4918, pp. 237–250. Springer, Heidelberg (2008). doi: 10.1007/978-3-540-79860-6_19
  16. 16.
    Lutz, C.: Interval-based temporal reasoning with general TBoxes. In: 17th International Joint Conference on Artificial Intelligence, pp. 89–94. Morgan Kaufmann (2001)Google Scholar
  17. 17.
    Artale, A., Franconi, E.: A temporal description logic for reasoning about actions and plans. J. Artif. Intell. Res. 9(1), 463–506 (1998)MathSciNetMATHGoogle Scholar
  18. 18.
    Liu, W., Xu, W., Wang, D., Liu, Z., Zhang, X.: A temporal description logic for reasoning about action in event. Inf. Technol. J. 11(9), 1211–1218 (2012). doi: 10.3923/itj.2012.1211.1218 CrossRefGoogle Scholar
  19. 19.
    Günsel, C., Wittmann, M.: Towards an implementation of the temporal DL \( \mathcal{T}\mathcal{L} \)-\( \mathcal{A}\mathcal{L}\mathcal{C} \). In: 2001 International Description Logics Workshop. Stanford University, Stanford (2001)Google Scholar
  20. 20.
    Artale, A., Lutz, C.: A correspondence between temporal description logics. J Appl. Non-Class. Log. 14(1–2), 209–233 (2012). doi: 10.3166/jancl.14.209-233 MATHGoogle Scholar
  21. 21.
    Baader, F., Lippmann, M.: Runtime verification using the temporal description logic \( \mathcal{ALC}{\text{-}}\mathcal{LTL} \) revisited. J. Appl. Log. 12(4), 584–613 (2014). doi: 10.1016/j.jal.2014.09.001
  22. 22.
    Kamide, N.: A compatible approach to temporal description logics. In: 23rd International Workshop on Description Logics, Waterloo (2010)Google Scholar
  23. 23.
    Artale, A., Franconi, E., Wolter, F., Zakharyaschev, M.: A temporal description logic for reasoning over conceptual schemas and queries. In: Flesca, S., Greco, S., Ianni, G., Leone, N. (eds.) Logics in Artificial Intelligence. LNCS, vol. 2424, pp. 98–110. Springer, Heidelberg (2002). doi: 10.1007/3-540-45757-7_9
  24. 24.
    Hu, K., Yu, X., Li, Z., Zhu, H.: The temporal description logic \( \mathcal{T}\mathcal{L} \)-\( \mathcal{S}\mathcal{I} \) and its decidability algorithm. In: 2010 International Conference on Computational Aspects of Social Networks, pp. 575–578. IEEE Press, New York (2010). doi: 10.1109/CASoN.2010.133
  25. 25.
    Artale, A., Kontchakov, R., Ryzhikov, V., Zakharyaschev, M.: Tailoring temporal description logics for reasoning over temporal conceptual models. In: Tinelli, C., Sofronie-Stokkermans, V. (eds.) Frontiers of Combining Systems. LNCS, vol. 6989, pp. 1–11. Springer, Heidelberg (2011). doi: 10.1007/978-3-642-24364-6_1
  26. 26.
    Milea, V., Frasincar, F., Kaymak, U.: tOWL: a temporal web ontology language. IEEE Trans. Syst. Man Cy. B 42(1), 268–281 (2012). doi: 10.1109/TSMCB.2011.2162582 CrossRefGoogle Scholar
  27. 27.
    Wang, Y., Chang, L., Li, F., Gu, T.: Verification of branch-time property based on dynamic description logic. In: Shi, Z., Wu, Z., Leake, D., Sattler, U. (eds.) Intelligent Information Processing. IFIP AICT, vol. 432, pp. 161–170. Springer, Heidelberg (2014). doi: 10.1007/978-3-662-44980-6_18
  28. 28.
    Gutiérrez-Basulto, V., Jung, J.C., Schneider, T.: Lightweight temporal description logics with rigid roles and restricted TBoxes. In: 24th International Joint Conference on Artificial Intelligence. AAAI Press, Palo Alto (2014)Google Scholar
  29. 29.
    Artale, A., Kontchakov, R., Ryzhikov, V., Zakharyaschev, M.: Interval temporal description logics. In: 28th International Workshop on Description Logics, Sun SITE Central Europe (2015)Google Scholar
  30. 30.
    Sanati, M.Y.: A metric interval-based temporal description logic. Ph.D. thesis, McMaster University, Hamilton (2015)Google Scholar
  31. 31.
    Artale, A., Calvanese, D., Kontchakov, R., Ryzhikov, V., Zakharyaschev, M.: Reasoning over extended ER models. In: 26th International Conference on Conceptual Modeling. LNCS, vol. 4801, pp. 277–292. Springer, Heidelberg (2007). doi: 10.1007/978-3-540-75563-0_20
  32. 32.
    Artale, A., Kontchakov, R., Ryzhikov, V., Zakharyaschev, M.: A cookbook for temporal conceptual data modelling with description logics. ACM Trans. Comput. Logic 15(3), article 25 (2014). doi: 10.1145/2629565
  33. 33.
    Sotnykova, A., Vangenot, C., Cullot, N., Bennacer, N., Aufaure, M.-A.: Semantic mappings in description logics for spatio-temporal database schema integration. In: Spaccapietra, S., Zimányi, E. (eds.) Journal on Data Semantics III. LNCS, vol. 3534, pp. 143–167. Springer, Heidelberg (2005). doi: 10.1007/11496168_7
  34. 34.
    Elleuch, N., Zarka, M., Ammar, A.B., Alimi, A.M.: A fuzzy ontology-based framework for reasoning in visual video content analysis and indexing. In: 11th International Workshop on Multimedia Data Mining. ACM, New York (2011). doi: 10.1145/2237827.2237828
  35. 35.
    Carrive, J., Pachet, F., Ronfard, R.: Using description logics for indexing audiovisual documents. In: 1998 International Workshop on Description Logics, Sun SITE Central Europe (1998)Google Scholar
  36. 36.
    Hammiche, S., Benbernou, S., Vakali, A.: A logic based approach for the multimedia data representation and retrieval. In: 7th IEEE International Symposium on Multimedia. IEEE Press, New York (2005). doi: 10.1109/ISM.2005.11
  37. 37.
    Bai, L., Lao, S., Jones, G.J.F., Smeaton, A.F.: Video semantic content analysis based on ontology. In: International Machine Vision and Image Processing Conference, pp. 117–124. IEEE Press, New York (2007). doi: 10.1109/IMVIP.2007.13
  38. 38.
    SanMiguel, J.C., Martinez, J.M., Garcia, Á.: An ontology for event detection and its application in surveillance video. In: 6th IEEE International Conference on Advanced Video and Signal Based Surveillance, pp. 220–225. IEEE Press, New York (2009). doi: 10.1109/AVSS.2009.28
  39. 39.
    Wang, X., Chang, L., Li, Z., Shi, Z.: A dynamic description logic based system for video event detection. Front. Electr. Electron. Eng. China 5(2), 137–142 (2010). doi: 10.1007/s11460-009-0078-y CrossRefGoogle Scholar
  40. 40.
    Helaoui, R., Riboni, D., Niepert, M., Bettini, C., Stuckenschmidt, H.: Towards activity recognition using probabilistic description logics. In: 26th AAAI Conference on Artificial Intelligence, Toronto, Ontario, Canada (2012)Google Scholar
  41. 41.
    Elbaşi, E.: Fuzzy logic-based scenario recognition from video sequences. J. Appl. Res. Technol. 11(5), 702–707 (2013). doi: 10.1016/S1665-6423(13)71578-5 CrossRefGoogle Scholar
  42. 42.
    Santofimia, M.J., Martinez-del-Rincon, J., Nebel, J.-C.: Episodic reasoning for vision-based human action recognition. Sci. World J. 2014, article 270171 (2014). doi: 10.1155/2014/270171
  43. 43.
    Sikos, L.F.: RDF-powered semantic video annotation tools with concept mapping to Linked Data for next-generation video indexing: a comprehensive review. Multimed. Tools Appl. 76(12), 14437–14460 (2016). doi: 10.1007/s11042-016-3705-7
  44. 44.
    Sikos, L.F.: Ontology-based structured video annotation for content-based video retrieval via spatiotemporal reasoning. In: Kwaśnicka, H., Jain, L.C. (eds.) Bridging the Semantic Gap in Image and Video Analysis. Intelligent Systems Reference Library. Springer, Cham (2017)Google Scholar
  45. 45.
    Ronfard, R.: Shot-level description and matching of video content. In: Jay Kuo, C.-C., Chang, S.-F., Gudivada, V.N. (eds.) Multimedia Storage and Archiving Systems II. The International Society for Optical Engineering, Bellingham (1997). doi: 10.1117/12.290366
  46. 46.
    Grutter, R., Bauer-Messmer, B.: Towards spatial reasoning in the semantic web: a hybrid knowledge representation system architecture. LNG&C, pp. 349–364. Springer, Heidelberg (2007). doi: 10.1007/978-3-540-72385-1_21
  47. 47.
    Batsakis, S., Petrakis, E.G.M., Tachmazidis, I., Antoniou, G.: Temporal representation and reasoning in OWL 2, Semantic Web – Interoperability, Usability, Applicability (2016)Google Scholar
  48. 48.
    Bobillo, F., Straccia, U.: Fuzzy ontology representation using OWL 2. Int. J. Approx. Reason, 52(7), 1073–1094 (2011). doi: 10.1016/j.ijar.2011.05.003 MathSciNetCrossRefGoogle Scholar
  49. 49.
    Allen, J.F.: Maintaining knowledge about temporal intervals. Commun. ACM 26(11), 832–843 (1983). doi: 10.1145/182.358434 CrossRefMATHGoogle Scholar

Copyright information

© Springer International Publishing AG 2018

Authors and Affiliations

  1. 1.Flinders UniversityAdelaideAustralia

Personalised recommendations