Skip to main content

Mixing Low-Level and Semantic Features for Image Interpretation

A Framework and a Simple Case Study

Part of the Lecture Notes in Computer Science book series (LNIP,volume 8926)

Abstract

Semantic Content-Based Image Retrieval (SCBIR) allows users to retrieve images via complex expressions of some ontological language describing a domain of interest. SCBIR adds some flexibility to the state-of-the-art methods for image retrieval, which support query either by keywords or by image examples. The price for this additional flexibility is the generation of a semantically rich description of the image content reflecting the ontology constraints. Generating these semantic interpretations is an open research problem. This paper contributes to this research line by proposing an approach for SCBIR based on the somehow natural idea that the interpretation of a picture is an (onto) logical model of an ontology that describes the domain of the picture. We implement this idea in an unsupervised method that jointly exploits the ontological constraints and the low-level features of the image. The preliminary evaluation, presented in the paper, shows promising results.

Keywords

  • Computer vision
  • Ontologies
  • Semantic image interpretation

References

  1. Abeel, T., Van de Peer, Y., Saeys, Y.: Javaml: A machine learning library. J. Mach. Learn. Res. 10, 931–934 (2009). http://dl.acm.org/citation.cfm?id=1577069.1577103

  2. Antanas, L., Frasconi, P., Costa, F., Tuytelaars, T., Raedt, L.D.: A relational kernel-based framework for hierarchical image understanding. In: Gimel’farb, G.L., Hancock, E.R., Imiya, A., Kuijper, A., Kudo, M., Omachi, S., Windeatt, T., Yamada, K. (eds.) SSPR/SPR. LNCS, vol. 7626, pp. 171–180. Springer, Heidelberg (2012)

    Google Scholar 

  3. Antanas, L., van Otterlo, M., Mogrovejo, J.O., Tuytelaars, T., Raedt, L.D.: A relational distance-based framework for hierarchical image understanding. In: Carmona, P.L., Sánchez, J.S., Fred, A.L.N. (eds.) ICPRAM (2), pp. 206–218. SciTePress (2012)

    Google Scholar 

  4. Atif, J., Hudelot, C., Bloch, I.: Explanatory reasoning for image understanding using formal concept analysis and description logics. IEEE Transactions on Systems, Man, and Cybernetics: Systems 44(5), 552–570 (2014)

    CrossRef  Google Scholar 

  5. Baader, F., Calvanese, D., McGuinness, D.L., Nardi, D., Patel-Schneider, P.F. (eds.): The Description Logic Handbook: Theory, Implementation, and Applications. Cambridge University Press, New York (2003)

    Google Scholar 

  6. Bannour, H., Hudelot, C.: Towards ontologies for image interpretation and annotation. In: Martinez, J.M. (ed.) 9th International Workshop on Content-Based Multimedia Indexing, CBMI 2011, June 13–15, Madrid, Spain, pp. 211–216. IEEE (2011)

    Google Scholar 

  7. Dasiopoulou, S., Kompatsiaris, I., Strintzis, M.G.: Applying fuzzy dls in the extraction of image semantics. J. Data Semantics 14, 105–132 (2009)

    CrossRef  Google Scholar 

  8. Diligenti, M., Gori, M., Maggini, M., Rigutini, L.: Bridging logic and kernel machines. Machine Learning 86(1), 57–88 (2012)

    CrossRef  MATH  MathSciNet  Google Scholar 

  9. Espinosa, S., Kaya, A., Möller, R.: Logical formalization of multimedia interpretation. In: Paliouras, G., Spyropoulos, C.D., Tsatsaronis, G. (eds.) Multimedia Information Extraction. LNCS, vol. 6050, pp. 110–133. Springer, Heidelberg (2011). http://dx.doi.org/10.1007/978-3-642-20795-2_5

  10. Fellbaum, C. (ed.): WordNet: an electronic lexical database. MIT Press (1998)

    Google Scholar 

  11. Haarslev, V., Hidde, K., Möller, R., Wessel, M.: The racerpro knowledge representation and reasoning system. Semantic Web Journal 3(3), 267–277 (2012)

    Google Scholar 

  12. Han, F., Zhu, S.C.: Bottom-up/top-down image parsing by attribute graph grammar. In: Tenth IEEE International Conference on Computer Vision, ICCV 2005, vol. 2, pp. 1778–1785 (October 2005)

    Google Scholar 

  13. Hobbs, J.R., Stickel, M.E., Appelt, D.E., Martin, P.: Interpretation as abduction. Artif. Intell. 63(1–2), 69–142 (1993). http://dx.doi.org/10.1016/0004-3702(93)90015-4

  14. Hudelot, C., Atif, J., Bloch, I.: Fuzzy spatial relation ontology for image interpretation. Fuzzy Sets and Systems 159(15), 1929–1951 (2008); from Knowledge Representation to Information Processing and Management Selected papers from the French Fuzzy Days (LFA 2006). http://www.sciencedirect.com/science/article/pii/S0165011408001012

  15. Kohonen, T.: The self-organizing map. Proceedings of the IEEE 78(9), 1464–1480 (1990)

    CrossRef  Google Scholar 

  16. Liu, H., Bao, H., Xu, D.: Concept vector for semantic similarity and relatedness based on wordnet structure. J. Syst. Softw. 85(2), 370–381 (2012). http://dx.doi.org/10.1016/j.jss.2011.08.029

  17. Liu, Y., Zhang, D., Lu, G., Ma, W.Y.: A survey of content-based image retrieval with high-level semantics. Pattern Recognition 40(1), 262–282 (2007). http://dx.doi.org/10.1016/j.patcog.2006.04.045

  18. Moller, R., Neumann, B., Wessel, M.: Towards computer vision with description logics: some recent progress. In: Proceedings of the Integration of Speech and Image Understanding, pp. 101–115 (1999)

    Google Scholar 

  19. Neumann, B., Mller, R.: On scene interpretation with description logics. Image and Vision Computing 26(1), 82–101 (2008) cognitive Vision-Special Issue. http://www.sciencedirect.com/science/article/pii/S0262885607001394

  20. Neumann, B., Weiss, T.: Navigating through logic-based scene models for high-level scene interpretations. In: Crowley, J.L., Piater, J.H., Vincze, M., Paletta, L. (eds.) ICVS 2003. LNCS, vol. 2626, pp. 212–222. Springer, Heidelberg (2003). http://dl.acm.org/citation.cfm?id=1765473.1765497

  21. Oliva, A., Torralba, A.: The role of context in object recognition. Trends in Cognitive Sciences 11(12), 520–527 (2007)

    CrossRef  Google Scholar 

  22. Peraldi, I.S.E., Kaya, A., Möller, R.: Formalizing multimedia interpretation based on abduction over description logic aboxes. In: Grau, B.C., Horrocks, I., Motik, B., Sattler, U. (eds.) Description Logics. CEUR Workshop Proceedings, vol. 477. CEUR-WS.org. (2009)

    Google Scholar 

  23. Reiter, R., Mackworth, A.K.: A logical framework for depiction and image interpretation. Artificial Intelligence 41(2), 125–155 (1989)

    CrossRef  MATH  MathSciNet  Google Scholar 

  24. Richardson, M., Domingos, P.: Markov logic networks. Machine Learning 62(1–2), 107–136 (2006)

    CrossRef  Google Scholar 

  25. Russell, B.C., Torralba, A., Murphy, K.P., Freeman, W.T.: Labelme: A database and web-based tool for image annotation. Int. J. Comput. Vision 77(1–3), 157–173 (2008). http://dx.doi.org/10.1007/s11263-007-0090-8

  26. Schroder, C., Neumann, B.: On the logics of image interpretation: model-construction in a formal knowledge-representation framework. In: Proceedings. of the Int. Conf. on Image Processing, vol. 1, pp. 785–788 (September 1996)

    Google Scholar 

  27. Sirin, E., Parsia, B., Grau, B.C., Kalyanpur, A., Katz, Y.: Pellet: A practical owl-dl reasoner. Web Semant. 5(2), 51–53 (2007). http://dx.doi.org/10.1016/j.websem.2007.03.004

  28. Smith, B., von Ehrenfels, C., Verlag, P.: Foundations of Gestalt theory. Philosophia Verlag Munich, Germany (1988)

    Google Scholar 

  29. Socher, R., Lin, C.C., Manning, C., Ng, A.Y.: Parsing natural scenes and natural language with recursive neural networks. In: Proceedings of the 28th International Conference on Machine Learning (ICML 2011), pp. 129–136 (2011)

    Google Scholar 

  30. Staruch, B., Staruch, B.: First order theories for partial models. Studia Logica 80(1), 105–120 (2005)

    CrossRef  MATH  MathSciNet  Google Scholar 

  31. Straccia, U.: Reasoning within fuzzy description logics. J. Artif. Intell. Res. (JAIR) 14, 137–166 (2001)

    MATH  MathSciNet  Google Scholar 

  32. Zlatoff, N., Tellez, B., Baskurt, A.: Image understanding and scene models: a generic framework integrating domain knowledge and gestalt theory. In: International Conference on Image Processing, ICIP 2004, vol. 4, pp. 2355–2358 (October 2004)

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Ivan Donadello .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2015 Springer International Publishing Switzerland

About this paper

Cite this paper

Donadello, I., Serafini, L. (2015). Mixing Low-Level and Semantic Features for Image Interpretation. In: Agapito, L., Bronstein, M., Rother, C. (eds) Computer Vision - ECCV 2014 Workshops. ECCV 2014. Lecture Notes in Computer Science(), vol 8926. Springer, Cham. https://doi.org/10.1007/978-3-319-16181-5_20

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-16181-5_20

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-16180-8

  • Online ISBN: 978-3-319-16181-5

  • eBook Packages: Computer ScienceComputer Science (R0)