A Relational Kernel-Based Framework for Hierarchical Image Understanding

  • Laura Antanas
  • Paolo Frasconi
  • Fabrizio Costa
  • Tinne Tuytelaars
  • Luc De Raedt
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7626)


While relational representations have been popular in early work on syntactic and structural pattern recognition, they are rarely used in contemporary approaches to computer vision due to their pure symbolic nature. The recent progress and successes in combining statistical learning principles with relational representations motivates us to reinvestigate the use of such representations. More specifically, we show that statistical relational learning can be successfully used for hierarchical image understanding. We employ kLog, a new logical and relational language for learning with kernels to detect objects at different levels in the hierarchy. The key advantage of kLog is that both appearance features and rich, contextual dependencies between parts in a scene can be integrated in a principled and interpretable way to obtain a qualitative representation of the problem. At each layer, qualitative spatial structures of parts in images are detected, classified and then employed one layer up the hierarchy to obtain higher-level semantic structures. We apply a four-layer hierarchy to street view images and successfully detect corners, windows, doors, and individual houses.


Ground Atom Contour Segment Graph Kernel Shape Grammar Object Layer 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Tuytelaars, T., Mikolajczyk, K.: Local invariant feature detectors: A survey. Foundations and Trends in Computer Graphics and Vision 3(3), 177–280 (2007)CrossRefGoogle Scholar
  2. 2.
    Hanson, A., Riseman, E.: Visions: A computer system for interpreting scenes. In: CVS, pp. 303–333 (1978)Google Scholar
  3. 3.
    De Raedt, L.: Logical and Relational Learning. Springer (2008)Google Scholar
  4. 4.
    Fu, K.: Syntactic methods in pattern recognition, vol. 112. Elsevier Science (1974)Google Scholar
  5. 5.
    Antanas, L., van Otterlo, M., Tuytelaars, T., Raedt, L.D., Oramas Mogrovejo, J.: A relational distance-based framework for hierarchical image understanding. In: ICPRAM, vol. (2), pp. 206–218 (2012)Google Scholar
  6. 6.
    Pearce, A.R., Caelli, T., Bischof, W.F.: Learning relational structures: Applications in computer vision. Applied Intelligence 4, 257–268 (1994)CrossRefGoogle Scholar
  7. 7.
    Getoor, L., Friedman, N., Koller, D., Taskar, B.: Learning probabilistic models of relational structure. In: ICML, pp. 170–177 (2001)Google Scholar
  8. 8.
    Frasconi, P., Costa, F., Raedt, L.D., Grave, K.D.: klog: A language for logical and relational learning with kernels. CoRR (2012)Google Scholar
  9. 9.
    Felzenszwalb, P., Girshick, R., McAllester, D., Ramanan, D.: Object detection with discriminatively trained part-based models. IEEE TPAMI 32(9), 1627–1645 (2010)CrossRefGoogle Scholar
  10. 10.
    Fergus, R., Perona, P., Zisserman, A.: Weakly supervised scale-invariant learning of models for visual recognition. IJCV 71(3), 273–303 (2007)CrossRefGoogle Scholar
  11. 11.
    Han, F., Zhu, S.: Bottom-up/top-down image parsing with attribute grammar. IEEE Transactions on Pattern Analysis and Machine Intelligence 31(1), 59–73 (2009)MathSciNetCrossRefGoogle Scholar
  12. 12.
    Zhu, L., Chen, Y., Lin, Y., Lin, C., Yuille, A.: Recursive segmentation and recognition templates for image parsing. IEEE TPAMI 34(2), 359–371 (2012)CrossRefGoogle Scholar
  13. 13.
    Girshick, R., Felzenszwalb, P., McAllester, D.: Object detection with grammar models. IEEE TPAMI 33(12) (2011)Google Scholar
  14. 14.
    Zhu, S.C., Mumford, D.: A stochastic grammar of images. Found. Trends. Comput. Graph. Vis. 2(4), 259–362 (2006)zbMATHCrossRefGoogle Scholar
  15. 15.
    Hartz, J.: Learning probabilistic structure graphs for classification and detection of object structures. In: ICMLA, pp. 5–11 (2009)Google Scholar
  16. 16.
    Zhao, P., Fang, T., Xiao, J., Zhang, H., Zhao, Q., Quan, L.: Rectilinear parsing of architecture in urban environment. In: CVPR, pp. 342–349 (2010)Google Scholar
  17. 17.
    Koutsourakis, P., Simon, L., Teboul, O., Tziritas, G., Paragios, N.: Single view reconstruction using shape grammars for urban environments. In: ICCV, pp. 1795–1802 (2009)Google Scholar
  18. 18.
    Terzic, K., Hotz, L., Sochman, J.: Interpreting structures in man-made scenes - combining low-level and high-level structure sources. In: ICAART, pp. 357–364 (2010)Google Scholar
  19. 19.
    Tuytelaars, T., Fritz, M., Saenko, K., Darrell, T.: The nbnn kernel. In: ICCV, pp. 1824–1831 (2011)Google Scholar
  20. 20.
    Antanas, L., Frasconi, P., Tuytelaars, T., De Raedt, L.: Employing relational languages for image understanding. In: IEEE Workshop on Kernels and Distances for Computer Vision, pp. 1–2 (2011)Google Scholar
  21. 21.
    Ferrari, V., Fevrier, L., Jurie, F., Schmid, C.: Groups of adjacent contour segments for object detection. TPAMI, 36–51 (2008)Google Scholar
  22. 22.
    Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: CVPR, pp. 886–893 (2005)Google Scholar
  23. 23.
    Garcia-Molina, H., Ullman, J.D., Widom, J.: Database Systems: The Complete Book, 2nd edn. Prentice Hall Press, Upper Saddle River (2008)Google Scholar
  24. 24.
    Costa, F., Grave, K.D.: Fast neighborhood subgraph pairwise distance kernel. In: ICML, pp. 255–262 (2010)Google Scholar
  25. 25.
    Haussler, D.: Convolution kernels on discrete structures. Technical Report UCSC-CRL-99-10, University of California at Santa Cruz (1999)Google Scholar
  26. 26.
    Fan, R.E., Chang, K.W., Hsieh, C.J., Wang, X.R., Lin, C.J.: Liblinear: A library for large linear classification. J. Mach. Learn. Res. 9, 1871–1874 (2008)zbMATHGoogle Scholar
  27. 27.
    Torralba, A., Murphy, K.P., Freeman, W.T.: Sharing features: Efficient boosting procedures for multiclass object detection. In: CVPR, pp. 762–769 (2004)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Laura Antanas
    • 1
  • Paolo Frasconi
    • 1
  • Fabrizio Costa
    • 1
  • Tinne Tuytelaars
    • 1
  • Luc De Raedt
    • 1
  1. 1.Katholieke Universiteit LeuvenBelgium

Personalised recommendations