Applying Fuzzy DLs in the Extraction of Image Semantics

  • Stamatia Dasiopoulou
  • Ioannis Kompatsiaris
  • Michael G. Strintzis
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5880)


Statistical learning approaches, bounded mainly to knowledge related to perceptual manifestations of semantics, fall short to adequately utilise the meaning and logical connotations pertaining to the extracted image semantics. Instigated by the Semantic Web, ontologies have appealed to a significant share of synergistic approaches towards the combined use of statistical learning and explicit semantics. While the relevant literature tends to disregard the uncertainty involved, and treats the extracted image descriptions as coherent, two valued propositions, this paper explores reasoning under uncertainty towards a more accurate and pragmatic handling of the underlying semantics. Using fuzzy DLs, the proposed reasoning framework captures the vagueness of the extracted image descriptions and accomplishes their semantic interpretation, while resolving inconsistencies rising from contradictory descriptions. To evaluate the proposed reasoning framework, an experimental implementation using the fuzzyDL Description Logic reasoner has been carried out. Experiments in the domain of outdoor images illustrate the added value, while outlining challenges to be further addressed.


Description Logic Object Level Image Description Reasoning Framework Scene Description 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Smeulders, A.W.M., Worring, M., Santini, S., Gupta, A., Jain, R.: Content-based image retrieval at the end of the early years. IEEE Trans. Pattern Anal. Mach. Intell. 22(12), 1349–1380 (2000)CrossRefGoogle Scholar
  2. 2.
    Chang, S.F.: The holy grail of content-based. IEEE MultiMedia 9(2), 6–10 (2002)CrossRefGoogle Scholar
  3. 3.
    Naphade, M., Huang, T.: Extracting semantics from audio-visual content: the final frontier in multimedia retrieval. IEEE Transactions on Neural Networks 13(4), 793–810 (2002)CrossRefGoogle Scholar
  4. 4.
    Hanjalic, A., Lienhart, R., Ma, W., Smith, J.: The holy grail of multimedia information retrieval: So close or yet so far away. IEEE Proceedings, Special Issue on Multimedia Information Retrieval 96(4), 541–547 (2008)Google Scholar
  5. 5.
    Burges, C.: A Tutorial on Support Vector Machines for Pattern Recognition. Data Mining and Knowledge Discovery 2(2), 121–167 (1998)CrossRefGoogle Scholar
  6. 6.
    Heckerman, D.: A tutorial on learning with bayesian networks. Learning in Graphical Models, 301–354 (1998)Google Scholar
  7. 7.
    Chapelle, O., Haffner, P., Vapnik, V.N.: Support vector machines for histogram-based image classification 10(5), 1055–1064 (1999)Google Scholar
  8. 8.
    Naphade, M., Huang, T.: A probabilistic framework for semantic video indexing, filtering, and retrieval. IEEE Transactions on Multimedia 3(1), 141–151 (2001)CrossRefGoogle Scholar
  9. 9.
    Assfalg, J., Bertini, M., Colombo, C., Bimbo, A.D.: Semantic annotation of sports videos. IEEE MultiMedia 9(2), 52–60 (2002)CrossRefGoogle Scholar
  10. 10.
    Christmas, W.J., Jaser, E., Messer, K., Kittler, J.: A multimedia system architecture for automatic annotation of sports videos. In: ICVS, pp. 513–522 (2003)Google Scholar
  11. 11.
    Town, C., Sinclair, D.: A self-referential perceptual inference framework for video interpretation. In: Crowley, J.L., Piater, J.H., Vincze, M., Paletta, L. (eds.) ICVS 2003. LNCS, vol. 2626, pp. 54–67. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  12. 12.
    Snoek, C., Worring, M., van Gemert, J., Geusebroek, J., Smeulders, A.: The challenge problem for automated detection of 101 semantic concepts in multimedia. In: Proc. 14th ACM International Conference on Multimedia, Santa Barbara, CA, USA, October 23-27, pp. 421–430 (2006)Google Scholar
  13. 13.
    Rao, A., Jain, R.: Knowledge representation and control in computer vision systems. IEEE Expert, 64–79 (1988)Google Scholar
  14. 14.
    Crevier, D., Lepage, R.: Knowledge-based image understanding systems: A survey. Computer Vision and Image Understanding 67, 161–185 (1997)CrossRefGoogle Scholar
  15. 15.
    Snoek, C., Huurnink, B., Hollink, L., Rijke, M., Schreiber, G., Worring, M.: Adding semantics to detectors for video retrieval. IEEE Transactions on Multimedia 9(5), 975–986 (2007)CrossRefGoogle Scholar
  16. 16.
    Horrocks, I., Patel-Schneider, P., van Harmelen, F.: From shiq and rdf to owl: the making of a web ontology language. J. Web Sem. 1(1), 7–26 (2003)Google Scholar
  17. 17.
    Horrocks, I., Patel-Schneider, P., Bechhofer, S., Tsarkov, D.: Owl rules: A proposal and prototype implementation. J. Web Semantics 3(1), 23–40 (2005)Google Scholar
  18. 18.
    Baader, F., Calvanese, D., McGuinness, D.L., Nardi, D., Patel-Schneider, P.F.: The description logic handbook: Theory, implementation, and applications. In: Description Logic Handbook. Cambridge University Press, Cambridge (2003)Google Scholar
  19. 19.
    Baader, F., Horrocks, I., Sattler, U.: Description logics as ontology languages for the semantic web. In: Mechanizing Mathematical Reasoning, pp. 228–248 (2005)Google Scholar
  20. 20.
    Hunter, J.: Adding Multimedia to the Semantic Web: Building an MPEG-7 Ontology. In: Proc. The First Semantic Web Working Symposium (SWWS), California, USA (July 2001)Google Scholar
  21. 21.
    Simou, N., Saathoff, C., Dasiopoulou, S., Spyrou, E., Voisine, N., Tzouvaras, V., Kompatsiaris, I., Avrithis, Y., Staab, S.: An ontology infrastructure for multimedia reasoning. In: Proc. International Workshop on Very Low Bitrate Video Coding (VLBV), Sardinia, Italy, September 15-16, pp. 51–60 (2005)Google Scholar
  22. 22.
    Arndt, R., Troncy, R., Staab, S., Hardman, L., Vacura, M.: COMM: Designing a well-founded multimedia ontology for the web. In: Aberer, K., Choi, K.-S., Noy, N., Allemang, D., Lee, K.-I., Nixon, L.J.B., Golbeck, J., Mika, P., Maynard, D., Mizoguchi, R., Schreiber, G., Cudré-Mauroux, P. (eds.) ASWC 2007 and ISWC 2007. LNCS, vol. 4825, pp. 30–43. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  23. 23.
    Dasiopoulou, S., Tzouvaras, V., Kompatsiaris, I., Strintzis, M.G.: Capturing mpeg-7 semantics. In: Proc. International Conference on Metadata and Semantics (MTSR), Corfu, Greece, October 11-12 (2007)Google Scholar
  24. 24.
    Troncy, R.: Integrating structure and semantics into audio-visual documents. In: Fensel, D., Sycara, K., Mylopoulos, J. (eds.) ISWC 2003. LNCS, vol. 2870, pp. 566–581. Springer, Heidelberg (2003)Google Scholar
  25. 25.
    Hunter, J., Drennan, J., Little, S.: Realizing the hydrogen economy through semantic web technologies. IEEE Intelligent Systems Journal - Special Issue on eScience 19, 40–47 (2004)Google Scholar
  26. 26.
    Petridis, K., Bloehdorn, S., Saathoff, C., Simou, N., Dasiopoulou, S., Tzouvaras, V., Handschuh, S., Avrithis, Y., Kompatsiaris, I., Staab, S.: Knowledge representation and semantic annotation of multimedia content. IEE Proceedings on Vision Image and Signal Processing, Special issue on Knowledge-Based Digital Media Processing 153, 255–262 (2006)Google Scholar
  27. 27.
    Little, S., Hunter, J.: Rules-by-example – A novel approach to semantic indexing and querying of images. In: McIlraith, S.A., Plexousakis, D., van Harmelen, F. (eds.) ISWC 2004. LNCS, vol. 3298, pp. 534–548. Springer, Heidelberg (2004)Google Scholar
  28. 28.
    Moller, R., Neumann, B., Wessel, M.: Towards computer vision with description logics: Some recent progress. In: Proc. Workshop on Integration of Speech and Image Understanding, Corfu, Greece, September 21, pp. 101–115 (1999)Google Scholar
  29. 29.
    Neumann, B., Moller, R.: On scene interpretation with description logics, FBI-B-257/04 (2004)Google Scholar
  30. 30.
    Bagdanov, A., Bertini, M., DelBimbo, A., Serra, G., Torniai, C.: Semantic annotation and retrieval of video events using multimedia ontologies. In: Proc. IEEE International Conference on Semantic Computing (ICSC), Irvine, CA, USA, pp. 713–720 (2007)Google Scholar
  31. 31.
    Espinosa, S., Kaya, A., Melzer, S., Möller, R., Wessel, M.: Multimedia interpretation as abduction. In: Proc. International Workshop on Description Logics (DL), Brixen-Bressanone, Italy, June 8-10, pp. 323–331 (2007)Google Scholar
  32. 32.
    Dasiopoulou, S., Mezaris, V., Kompatsiaris, I., Papastathis, V., Strintzis, M.: Knowledge-assisted semantic video object detection. IEEE Trans. Circuits Syst. Video Techn. 15(10), 1210–1224 (2005)CrossRefGoogle Scholar
  33. 33.
    Dasiopoulou, S., Kompatsiaris, I., Strintzis, M.: Using fuzzy dLs to enhance semantic image analysis. In: Duke, D., Hardman, L., Hauptmann, A., Paulus, D., Staab, S. (eds.) SAMT 2008. LNCS, vol. 5392, pp. 31–46. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  34. 34.
    Maron, O., Ratan, A.: Multiple-instance learning for natural scene classification. In: Proc. 15th International Conference on Machine Learning (ICML), Madison, Wisconson, USA, July 24-27, pp. 341–349 (1998)Google Scholar
  35. 35.
    Vailaya, A., Figueiredo, M., Jain, A., Zhang, H.: Image classification for content-based indexing. IEEE Transactions on Image Processing 10(1), 117–130 (2001)zbMATHCrossRefGoogle Scholar
  36. 36.
    Barnard, K., Duygulu, P., Forsyth, D., de Freitas, N., Blei, D., Jordan, M.: Matching words and pictures. Journal of Machine Learning Research 3, 1107–1135 (2003)zbMATHCrossRefGoogle Scholar
  37. 37.
    Hauptmann, A., Yan, R., Lin, W.H., Christel, M., Wactlar, H.: Can high-level concepts fill the semantic gap in video retrieval? a case study with broadcast news. IEEE Transactions on Multimedia 9(5), 958–966 (2007)CrossRefGoogle Scholar
  38. 38.
    Niemann, H., Sagerer, G., Schröder, S., Kummert, F.: Ernest: A semantic network system for pattern understanding. IEEE Trans. Pattern Anal. Mach. Intell. 12(9), 883–905 (1990)CrossRefGoogle Scholar
  39. 39.
    Reiter, R., Mackworth, A.K.: A logical framework for depiction and image interpretation. Artif. Intell. 41(2), 125–155 (1989)zbMATHCrossRefMathSciNetGoogle Scholar
  40. 40.
    Russ, T., MacGregor, R., Salemi, B., Price, K., Nevatia, R.: Veil: Combining semantic knowledge with image understanding. In: ARPA Image Understanding Workshop, Palm Springs, CA, USA, February 12-17 (1996)Google Scholar
  41. 41.
    Rabiner, L., Juang, B.: An introduction to hidden markov models. IEEE ASSP Magazine, [see also IEEE Signal Processing Magazine] 3(1), 4–16 (1986)Google Scholar
  42. 42.
    Dubois, D., Prade, H.: Possibility theory, probability theory and multiple-valued logics: A clarification. Annals of Mathematics and Artificail Intelligence 32(1-4), 35–66 (2001)CrossRefMathSciNetGoogle Scholar
  43. 43.
    Zadeh, L.: Fuzzy sets. Information and Control 8(32), 338–353 (1965)zbMATHCrossRefMathSciNetGoogle Scholar
  44. 44.
    Klir, G., Yuan, B.: Fuzzy sets and fuzzy logic: Theory and applications. Prentice-Hall, Englewood Cliffs (1995)zbMATHGoogle Scholar
  45. 45.
    Yen, J.: Generalizing term subsumption languages to fuzzy logic. In: Proc. 12th International Joint Conference on Artificial Intelligence (IJCAI), Sydney, Australia, August 24-30, pp. 472–477 (1991)Google Scholar
  46. 46.
    Straccia, U.: A fuzzy description logic. In: Proc. International Conference on Artificial Intelligence and 10th Innovative Applications of Artificial Intelligence Conference (AAAI/IAAI), Madison, Wisconsin, July 26-30, pp. 594–599 (1998)Google Scholar
  47. 47.
    Straccia, U.: Reasoning within fuzzy description logics. J. Artif. Intell. Res. (JAIR) 14, 137–166 (2001)zbMATHMathSciNetGoogle Scholar
  48. 48.
    Straccia, U.: Transforming fuzzy description logics into classical description logics. In: Alferes, J.J., Leite, J. (eds.) JELIA 2004. LNCS (LNAI), vol. 3229, pp. 385–399. Springer, Heidelberg (2004)Google Scholar
  49. 49.
    Stoilos, G., Stamou, G., Tzouvaras, V., Pan, J., Horrocks, I.: The fuzzy description logic f-SHIN. In: International Workshop on Uncertainty Reasoning For the Semantic Web (URSW), Galway, Ireland, November 7, pp. 67–76 (2005)Google Scholar
  50. 50.
    Stoilos, G., Stamou, G., Pan, J.: Handling imprecise knowledge with fuzzy description logic. In: Proc. International Workshop on Description Logics (DL), Lake District, UK, pp. 119–127 (2006)Google Scholar
  51. 51.
    Bell, D., Qi, G., Liu, W.: Approaches to inconsistency handling in description-logic based ontologies. In: Meersman, R., Tari, Z., Herrero, P. (eds.) OTM-WS 2007, Part II. LNCS, vol. 4806, pp. 1303–1311. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  52. 52.
    Lam, J., Sleeman, D., Pan, J., Vasconcelos, W.: A fine-grained approach to resolving unsatisfiable ontologies. In: Spaccapietra, S. (ed.) Journal on Data Semantics X. LNCS, vol. 4900, pp. 62–95. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  53. 53.
    Straccia, U.: A fuzzy description logic for the semantic web. In: Sanchez, E. (ed.) Fuzzy Logic and the Semantic Web. Capturing Intelligence, pp. 73–90. Elsevier, Amsterdam (2006)Google Scholar
  54. 54.
    Stoilos, G., Stamou, G., Pan, J., Tzouvaras, V., Horrocks, I.: Reasoning with very expressive fuzzy description logics. J. Artif. Intell. Res. (JAIR) 30, 273–320 (2007)zbMATHMathSciNetGoogle Scholar
  55. 55.
    Simou, N., Athanasiadis, T., Tzouvaras, V., Kollias, S.: Multimedia reasoning with f-shin. In: 2nd International Workshop on Semantic Media Adaptation and Personalization (SMAP), London, UK, pp. 413–420 (2007)Google Scholar
  56. 56.
    Bobillo, F., Straccia, U.: fuzzydl: An expressive fuzzy description logic reasoner. In: Proc. International Conference on Fuzzy Systems (FUZZ), Hong Kong, June 1-6, pp. 923–930. IEEE Computer Society, Los Alamitos (2008)Google Scholar
  57. 57.
    Papadopoulos, G.T., Mylonas, P., Mezaris, V., Avrithis, Y., Kompatsiaris, I.: Knowledge-assisted image analysis based on context and spatial optimization (2006)Google Scholar
  58. 58.
    Umberto, S., Giulio, V.: Dlmedia: an ontology mediated multimedia information retrieval system. In: Proc. International Workshop on Description Logics (DL), Brixen-Bressanone, Italy, June 8-10, pp. 467–475Google Scholar
  59. 59.
    Neumann, B., Weiss, T.: Navigating through logic-based scene models for high-level scene interpretations. In: Crowley, J.L., Piater, J.H., Vincze, M., Paletta, L. (eds.) ICVS 2003. LNCS, vol. 2626, pp. 212–222. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  60. 60.
    Schober, J.P., Hermes, T., Herzog, O.: Content-based image retrieval by ontology-based object recognition. In: Proc. KI 2004 Workshop on Applications of Description Logics (ADL), Ulm Germany, September 24, pp. 1–10 (2004)Google Scholar
  61. 61.
    Hu, B., Dasmahapatra, S., Lewis, P., Shadbolt, N.: Ontology-based medical image annotation with description logics. In: Proc. 15th IEEE International Conference on Tools with Artificial Intelligence (ICTAI), Sacramento, California, USA, November 3-5, pp. 77–83 (2002)Google Scholar
  62. 62.
    Dasiopoulou, S., Heinecke, J., Saathoff, C., Strintzis, M.: Multimedia reasoning with natural language support. In: Proc. IEEE International Conference on Semantic Computing (ICSC), Irvine, CA, USA, September 17-19 (2007)Google Scholar
  63. 63.
    Meghini, C., Sebastiani, F., Straccia, U.: A model of multimedia information retrieval. J. ACM 48(5), 909–970 (2001)CrossRefMathSciNetGoogle Scholar
  64. 64.
    Mylonas, P., Vallet, D., Castells, P., Fernandez, M., Avrithis, Y.: Personalized information retrieval based on context and ontological knowledge 23(1), 73–100 (March 2008)Google Scholar
  65. 65.
    Leger, A., Heinecke, J., Nixon, L., Shvaiko, P., Charlet, J., Hobson, P., Goasdoue, F.: Semantic web take-off in a european industry perspective. In: Garcia, R. (ed.) Semantic Web for Business: Cases and Applications, ch. 1, pp. 1–29. IGI Global (2008)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2009

Authors and Affiliations

  • Stamatia Dasiopoulou
    • 1
  • Ioannis Kompatsiaris
    • 1
  • Michael G. Strintzis
    • 1
    • 2
  1. 1.Centre for Research and Technology HellasInformatics and Telematics InstituteThessalonikiGreece
  2. 2.Information Processing Laboratory, Electrical and Computer Engineering DepartmentAristotle University of ThessalonikiGreece

Personalised recommendations