Multimedia Tools and Applications

, Volume 75, Issue 10, pp 5719–5750 | Cite as

Fuzzy reasoning framework to improve semantic video interpretation

  • Mohamed ZarkaEmail author
  • Anis Ben Ammar
  • Adel M. Alimi


A video retrieval system user hopes to find relevant information when the proposed queries are ambiguous. The retrieval process based on detecting concepts remains ineffective in such a situation. Potential relationships between concepts have been shown as a valuable knowledge resource that can enhance the retrieval effectiveness, even for ambiguous queries. Recent researches in multimedia retrieval have focused on ontology modeling as a common framework to manage knowledge. Handling these ontologies has to cope with issues related to generic knowledge management and processing scalability. Considering these issues, we suggest a context-based fuzzy ontology framework for video content analysis and indexing. In this paper, we focused on the way in which we modeled our fuzzy ontology: First, we populate automatically the generated ontology by gathering various available video annotation datasets. Then, the ontology content was used to infer enhanced video semantic interpretation. Finally, considering user feedback, the content of the ontology was improved. Experimental results showed that our approach achieves the goal of scalability while at the same time allowing better video content semantic interpretation.


Multimedia retrieval Ontology Video indexing 



The authors would like to acknowledge the financial support of this work by grants from General Direction of Scientific Research (DGRST), Tunisia, under the ARUB program.


  1. 1.
    (2006). LSCOM Lexicon Definitions Version, Annotations, Version 1.0., Tech. rep. Columbia UniversityGoogle Scholar
  2. 2.
    Adami N, Bugatti A, Leonardi R, Migliorati P (2001) Low level processing of audio and video information for extracting the semantics of content. In: 2001 IEEE Fourth Workshop on Multimedia Signal Processing, pp 607–612Google Scholar
  3. 3.
    Ayache S (2007) Indexation de documents vidéos par concepts par fusion de caractristiques audio, image et texte, Ph.D. thesis, Institut National Polytechnique de GrenobleGoogle Scholar
  4. 4.
    Ayache S (2008) Video Corpus Annotation using Active Learning. In: European Conference on Information Retrieval (ECIR). Glasgow, Scotland, pp 187–198Google Scholar
  5. 5.
    Baader F, Calvanese D, McGuinness D L, Nardi D, Patel-Schneider PF (eds) (2003) The description logic handbook: theory, implementation, and applications. Cambridge University Press, New YorkGoogle Scholar
  6. 6.
    Baghdadi S, Gravier G, Demarty C, Gros P (2008) Structure learning in a bayesian network-based video indexing framework. In: 2008 IEEE International Conference on Multimedia and Expo, pp. 677–680Google Scholar
  7. 7.
    Bannour H, Hudelot C (2013) Building and using fuzzy multimedia ontologies for semantic image annotation. Multimed Tools Appl:1–35Google Scholar
  8. 8.
    Benitez A, Chang SF (2003) Image classification using multimedia knowledge networks. In: 2003. ICIP 2003. Proceedings. 2003 International Conference on Image Processing, vol. 3, pp. III–613–16 vol.2Google Scholar
  9. 9.
    Bobillo F, Delgado M, Gmez-Romero J, Straccia U (2012) Joining gödel and zadeh fuzzy logics in fuzzy description logics, vol 20Google Scholar
  10. 10.
    Bosko B (1990) Fuzziness vs. probability. Int J Gen Syst 17(2-3):211–240CrossRefzbMATHGoogle Scholar
  11. 11.
    Brilhault A (2009) Indexation et recherche par le contenu de documents vidéos, Joseph Fourier UniversityGoogle Scholar
  12. 12.
    Calegari S, Ciucci D (2007) Fuzzy ontology, fuzzy description logics and fuzzy-owl. In: Proceedings of the 7th international workshop on Fuzzy Logic and Applications: Applications of Fuzzy Sets Theory, WILF ’07. Springer-Verlag, Berlin, Heidelberg, pp 118–126Google Scholar
  13. 13.
    Chattopadhyay C, Maurya A (2013) Genre-specific modeling of visual features for efficient content based video shot classification and retrieval. Int J Multimed Inf Retr 2(4):289–297. doi: 10.1007/s13735-013-0034-8 CrossRefGoogle Scholar
  14. 14.
    Cheng Y, Xiong Y (2012) Research on model of ontology-based semantic information retrieval. In: Jin D, Lin S (eds) Advances in Multimedia, Software Engineering and Computing Vol.1, vol 128. Springer , Berlin Heidelberg, pp 271–276Google Scholar
  15. 15.
    Dasiopoulou S, Giannakidou E, Litos G, Malasioti P, Kompatsiaris Y (2011) A survey of semantic image and video annotation tools. In: Paliouras G, Spyropoulos C , Tsatsaronis G (eds) Knowledge-Driven Multimedia Information Extraction and Ontology Evolution, vol 6050. Springer, Berlin Heidelberg, pp 196–239CrossRefGoogle Scholar
  16. 16.
    Dean J (2009) Challenges in building large-scale information retrieval systems: invited talk. In: Proceedings of the Second ACM International Conference on Web Search and Data Mining, WSDM ’09, New York, pp 1–1Google Scholar
  17. 17.
    DeMenthon D, Megret R (2002) Spatio-Temporal Segmentation of Video by Hierarchical Mean Shift Analysis. Tech. Rep. LAMP-TR-090, CAR-TR-978, CS-TR-4388, UMIACS-TR-2002-68, University of Maryland, College ParkGoogle Scholar
  18. 18.
    Dentler K, Cornet R, Ten Teije A, De Keizer N (2011) Comparison of reasoners for large ontologies in the owl 2 el profile. Semant. web 2(2):71–87Google Scholar
  19. 19.
    Du Y, Chen F, Xu W, Zhang W (2006) Interacting activity recognition using hierarchical durational-state dynamic bayesian network. In: Zhuang Y, Yang S Q, Rui Y, He Q (eds) Advances in Multimedia Information Processing - PCM 2006, vol 4261. Springer , Berlin Heidelberg, pp 185–192CrossRefGoogle Scholar
  20. 20.
    Egozi O, Markovitch S, Gabrilovich E (2011) Concept-based information retrieval using explicit semantic analysis. ACM Trans Inf Syst 29(2):8:1–8:34CrossRefGoogle Scholar
  21. 21.
    Elleuch N, Ben Ammar A, Alimi A M (2010) A generic system for semantic video indexing by visual concept. In: 2010 5th International Symposium on I/V Communications and Mobile Network (ISVC)Google Scholar
  22. 22.
    Elleuch N, Zarka M, Ben Ammar A, Alimi MA (2011) A fuzzy ontology: based framework for reasoning in visual video content analysis and indexing. In: Proceedings of the Eleventh International Workshop on Multimedia Data Mining, MDMKDD ’11. New York, pp 1–1Google Scholar
  23. 23.
    Elleuch N, Zarka M, Feki I, Ben Ammar A, Alimi MA (2010) Regimvid at trecvid 2010: Semantic indexing, TRECVID. 2010Google Scholar
  24. 24.
    Faria C, Girardi R (2011) An information extraction process for semi-automatic ontology population. In: Corchado E, Snel V, Sedano J , Hassanien A , Calvo J , lzak D (eds) Soft Computing Models in Industrial and Environmental Applications, 6th International Conference SOCO 2011, Advances in Intelligent and Soft Computing, vol 87, pp 319–328. Springer , Berlin HeidelbergGoogle Scholar
  25. 25.
    Fellbaum C (2010) Wordnet. In: Poli R, Healy M, Kameas A (eds) Theory and Applications of Ontology: Computer Applications. Springer , Netherlands, pp 231–243CrossRefGoogle Scholar
  26. 26.
    Fernndez-López M (1999) Overview of methodologies for building ontologies. In: Proceedings of the IJCAI-99 Workshop on Ontologies and Problem Solving Methods (KRR5) Stockholm, Sweden, August 2, 1999Google Scholar
  27. 27.
    Fu G, Jones C, Abdelmoty A (2005) Ontology-based spatial query expansion in information retrieval. In: Meersman R, Tari Z (eds) On the Move to Meaningful Internet Systems 2005: CoopIS, DOA, and ODBASE, vol 3761. Springer, Berlin Heidelberg, pp 1466–1482CrossRefGoogle Scholar
  28. 28.
    Gargouri F, Jaziri W (2010) Ontology Theory, Management and Design: Advanced Tools and Models. Premier Reference Source. Information Science ReferenceGoogle Scholar
  29. 29.
    Horrocks I (2012) Semantics ; scalability: Journal of Zhejiang University - Science C 13(4) 241–244Google Scholar
  30. 30.
    Huang YF, Wang SH (2012) Movie genre classification using svm with audio and video features. In: Huang R, Ghorbani A, Pasi G, Yamaguchi T, Yen N, Jin B (eds) Active Media Technology, vol 7669. Springer , Berlin Heidelberg, pp 1–10Google Scholar
  31. 31.
    Jiang Y G, Wang J, Chang S F, Ngo C W (2009) Domain adaptive semantic diffusion for large scale context-based video annotation. In: IEEE 12th International Conference on Computer Vision, pp 1420 –1427Google Scholar
  32. 32.
    Kara S (2010) An ontology-absed retrieval system using semantic indexing, Ph.D. thesis, Middle East Technical UniversityGoogle Scholar
  33. 33.
    Ksentini N, Zarka M, Ben Ammar A, Alimi MA (2012) Toward an assisted context based collaborative annotation. In: 10th International Workshop on Content-Based Multimedia Indexing (CBMI), 2012, pp 1 –6Google Scholar
  34. 34.
    Ksibi A, Ben Ammar A, Ben Amar C (2014) Adaptive diversification for tag-based social image retrieval. IJMIR 3(1):29–39Google Scholar
  35. 35.
    Ksibi A, Dammak M, Ben Ammar A, Mejdoub M, Ben Amar C (2012) Flickr-based semantic context to refine automatic photo annotation. In: Image Processing Theory, Tools and Applications (IPTA), 2012 3rd International Conference on, pp 377–382Google Scholar
  36. 36.
    Kumar S, Rana RK, Singh P (2012) Ontology based semantic indexing approach for information retrieval system, vol 49, pp 14–18. Published by Foundation of Computer Science, New York, USAGoogle Scholar
  37. 37.
    Leite M, Ricarte I (2008) A framework for information retrieval based on fuzzy relations and multiple ontologies. In: Geffner H, Prada R, Machado Alexandre I, David N (eds) Advances in Artificial Intelligence IBERAMIA 2008, vol 5290. Springer , Berlin Heidelberg, pp 292–301CrossRefGoogle Scholar
  38. 38.
    Li Z, Ramani K (2007) Ontology-based design information extraction and retrieval. AI EDAM 21:137–154Google Scholar
  39. 39.
    Mukesh R, Penchala S, Ingale A (2013) Ontology based zone indexing using information retrieval systems. In: Unnikrishnan S, Surve S , Bhoir D (eds) Advances in Computing, Communication, and Control, vol 361. Springer , Berlin Heidelberg, pp 181–186CrossRefGoogle Scholar
  40. 40.
    Muneesawang P, Zhang N, Guan L (2014) Scalable video genre classification and event detection. In: Multimedia Database Retrieval, Multimedia Systems and Applications, pp 247–278. Springer International PublishingGoogle Scholar
  41. 41.
    Mustafa J, Khan S, Latif K (2008) Ontology based semantic information retrieval. In: 2008 IS ’08. 4th International IEEE Conference Intelligent Systems, vol 3, pp 22–14–22–19Google Scholar
  42. 42.
    Mylonas P, Athanasiadis T, Wallace M, Avrithis Y, Kollias S (2008) Semantic representation of multimedia content: Knowledge representation and semantic indexing. Multimed Tools Appl 39(3):293–327CrossRefGoogle Scholar
  43. 43.
    Mylonas P, Spyrou E, Avrithis Y, Kollias S (2009) Using visual context and region semantics for high-level concept detection. Multimed, IEEE Trans on 11(2):229–243CrossRefGoogle Scholar
  44. 44.
    Nguyen C T (2010) Bridging semantic gaps in information retrieval: Context-based approaches. In: VLDB doctoral workshop, Singapore 2010Google Scholar
  45. 45.
    Nikolopoulos S, Papadopoulos G, Kompatsiaris I, Patras I (2009) An evidence-driven probabilistic inference framework for semantic image understanding. In: Perner P (ed) Machine Learning and Data Mining in Pattern Recognition, vol 5632. Springer , Berlin Heidelberg, pp 525–539CrossRefGoogle Scholar
  46. 46.
    Nikolopoulos S, Papadopoulos G T, Kompatsiaris I, Patras I (2011) Evidence-driven image interpretation by combining implicit and explicit knowledge in a bayesian network, IEEE Transactions on Systems, Man, and Cybernetics, Part B 41(5),1366–1381Google Scholar
  47. 47.
    Noy NF, Mcguinness D L (2001) Ontology development 101: A guide to creating your first ontology. Tech. Rep. KSL-01-05, Stanford Knowledge Systems LaboratoryGoogle Scholar
  48. 48.
    Over P, Awad G, Michel M, Fiscus J, Sanders G, Kraaij W, Smeaton AF, Quenot G (2013) Trecvid 2013 – an overview of the goals, tasks, data, evaluation mechanisms and metrics. In: Proceedings of TRECVID 2013. NIST USAGoogle Scholar
  49. 49.
    Paliouras G, Spyropoulos C, Tsatsaronis G (2011) Bootstrapping ontology evolution with multimedia information extraction. In: Paliouras G , Spyropoulos C , Tsatsaronis G (eds) Knowledge-Driven Multimedia Information Extraction and Ontology Evolution, vol 6050 . Springer, Berlin Heidelberg, pp 1–17CrossRefGoogle Scholar
  50. 50.
    Paliouras G, Spyropoulos CD, Tsatsaronis G (2011) Bootstrapping ontology evolution with multimedia information extraction. Lect Notes in Comput Sci 6050Google Scholar
  51. 51.
    Paliouras G, Spyropoulos C D, Tsatsaronis G (2011) Knowledge-Driven Multimedia Information Extraction and Ontology Evolution - Bridging the Semantic Gap, vol 6050. SpringerGoogle Scholar
  52. 52.
    Park S, Aggarwal J (2004) A hierarchical bayesian network for event recognition of human actions and interactions. Multimedia Systems 10(2):164–179CrossRefGoogle Scholar
  53. 53.
    Perpetual Coutinho F, Asnani K, Amos Caeiro D (2012) Context based information retrieval. International Journal of Advanced Research in Computer Science and Electronics Engineering (IJARCSEE) 1(7)Google Scholar
  54. 54.
    Petasis G, Karkaletsis V, Paliouras G, Krithara A, Zavitsanos E (2011) Ontology population and enrichment: State of the artGoogle Scholar
  55. 55.
    Petersohn C (2004) Fraunhofer hhi at trecvid 2004: Shot boundary detection system. In: TREC Video Retrieval Evaluation Online Proceedings, TRECVIDGoogle Scholar
  56. 56.
    Petridis K, Bloehdorn S, Saathoff C, Simou N, Dasiopoulou S, Tzouvaras V, Handschuh S, Avrithis Y, Kompatsiaris Y, Staab S (2006) Knowledge representation and semantic annotation of multimedia content. Vision, Image and Signal Processing. IEE Proceedings - 153(3):255–262Google Scholar
  57. 57.
    Rodrguez-Garca M, Valencia-Garca R , Garca-Snchez F (2012) An ontology evolution-based framework for semantic information retrieval. In: Herrero P, Panetto H, Meersman R, Dillon T (eds) On the Move to Meaningful Internet Systems: OTM 2012 Workshops, vol 7567 . Springer , Berlin Heidelberg, pp 163–172CrossRefGoogle Scholar
  58. 58.
    Romero AA, Grau BC, Horrocks I, Jiménez-Ruiz E (2013) More: a modular owl reasoner for ontology classification. In: Bail S , Glimm B, Gonçalves R S, Jiménez-Ruiz E, Kazakov Y , Matentzoglu N, Parsia B (eds) ORE, CEUR Workshop Proceedings, vol 1015, pp 61–67. CEUR-WS.orgGoogle Scholar
  59. 59.
    Rozilawati binti D, Masaki A (2011) Ontology based approach for classifying biomedical text abstracts. International Journal of Data Engineering 2(1)Google Scholar
  60. 60.
    Sanjaa B, Tsoozol P (2007) Fuzzy and probability. In: Strategic Technology, 2007. IFOST 2007. International Forum on, pp. 141–143Google Scholar
  61. 61.
    Sari RF, Ayuningtyas N (2010) Implementation of web ontology and semantic application for electronic journal citation system. Journal Of Emerging Technologies in Web Intelligence 2:34–41CrossRefGoogle Scholar
  62. 62.
    Simou N, Kollias S (2007) Fire: A fuzzy reasoning engine for impecise knowledge. K-Space PhD Students Workshop, Berlin, Germany, 14 September 2007Google Scholar
  63. 63.
    Smeaton AF, Over P, Kraaij W (2006) Evaluation campaigns and trecvid. In: Proceedings of the 8th ACM international workshop on Multimedia information retrieval, MIR ’06, pp 321–330, ACM, New York, NY, USAGoogle Scholar
  64. 64.
    Snoek CGM, Worring M (2009) Concept-based video retrieval. Foundations and Trends in Information Retrieval 2(4):215–322CrossRefGoogle Scholar
  65. 65.
    Staab S, Studer R (2009) Handbook on Ontologies, 2nd edn. Springer Publishing Company, IncorporatedGoogle Scholar
  66. 66.
    Stoilos G, Stamou GB, Tzouvaras V, Pan JZ, Horrocks I (2005) The fuzzy description logic f-shin. International Workshop on Uncertainty Reasoning For the Semantic Web (2005)Google Scholar
  67. 67.
    Thomee B, Popescu A (2012) Overview of the imageclef 2012 flickr photo annotation and retrieval task. In: Forner P , Karlgren J, Womser-Hacker C (eds) CLEF (Online Working Notes/Labs/Workshop)Google Scholar
  68. 68.
    Vallet D, Castells P, Fernandez M, Mylonas P, Avrithis Y (2007) Personalized content retrieval in context using ontological knowledge. Circuits and Systems for Video Technology. IEEE Transactions on 17(3):336–346Google Scholar
  69. 69.
    Wu F, Wu G, Fu X (2008) Design and implementation of ontology-based query expansion for information retrieval. In: Xu L, Tjoa A, Chaudhry S (eds) Research and Practical Issues of Enterprise Information Systems II, vol 254, pp 293–298. Springer US,Google Scholar
  70. 70.
    Wu J, Worring M (2012) Efficient genre-specific semantic video indexing. Multimedia, IEEE Transactions on 14(2 ):291–302 . doi: 10.1109/TMM.2011.2174969 CrossRefGoogle Scholar
  71. 71.
    Zadeh L (2014) Fuzzy set theory and probability theory: What is the relationship? In: Lovric M (ed) International Encyclopedia of Statistical Science. Springer , Berlin Heidelberg, pp 563–566Google Scholar
  72. 72.
    Zarka M, Ben Ammar A, Alimi M A (2011) Multimodale fuzzy fusion for semantic video indexing. In: IEEE Symposium Series in Computational Intelligence 2011 - CIMSIVPGoogle Scholar
  73. 73.
    Zhai J, Li M, Li J (2012) Semantic information retrieval based on rdf and fuzzy ontology for university scientific research management. In: Luo J (ed) Affective Computing and Intelligent Interaction, vol 137. Springer , Berlin Heidelberg, pp 661–668CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2015

Authors and Affiliations

  1. 1.REGIM: Reaserch Groups on Intelligent MachinesUniversity of Sfax, ENISSfaxTunisia

Personalised recommendations