Images Annotation Extension Based on User Feedback

  • Abdessalem BouzaieniEmail author
  • Salvatore Tabbone
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10617)


In this paper, we propose a probabilistic graphical model for images annotation extension. The aim is to extend the annotations of a small subset of images to a whole dataset. Therefore, this subset is used to learn the parameters of our model, which is based on multinomial and Gaussian mixture distributions. Our model allows combining efficiently visual and textual characteristics. Since the performance of our system depends on the quality of the learning, we integrate the user in the loop to improve the annotation quality and minimize the laborious manual annotation effort at three levels. The first level is related to the learning set. We perform a kind of learning in learning. More precisely, we propose a way to annotate semi-automatically images used in the learning. We introduce an iterative loop where annotations are automatically extended and some corrected manually by the user. In this way we reduce the tedious effort of manual annotation. In the second level, after the annotation extension and during a retrieval step a user can correct or add labels to some images. These images with their new labels are introduced progressively to the system and used to relearn incrementally the model. In the third level, we propose an active learning of our model to select the most informative data to improve the quality of learning and reduce manual effort.


  1. 1.
    Naphade, M.R., Lin, C.Y., Smith, J.R., Tseng, B.L., Basu, S.: Learning to annotate video databases. In Electronic Imaging, pp. 264–275 (2002)Google Scholar
  2. 2.
    Zhang, C., Chen, T.: Annotating retrieval database with active learning. In: International Conference on Image Processing (2003)Google Scholar
  3. 3.
    Wenyin, L., Dumais, S., Sun, Y., Zhang, H., Czerwinski, M., Field, B.: Semi-automatic image annotation. In: Proceedings of Human-Computer Interaction-Interact, pp. 326–333 (2001)Google Scholar
  4. 4.
    Vondrick, C., Ramanan, D.: Video annotation and tracking with active learning. In: Advances in Neural Information Processing Systems, pp. 28–36 (2011)Google Scholar
  5. 5.
    Byun, B., Lee, C.H.: An incremental learning framework combining sample confidence and discrimination with an application to automatic image annotation. In: ICIP, pp. 1441–1444 (2009)Google Scholar
  6. 6.
    Sychay, G., Chang, E., Goh, K.: Effective image annotation via active learning. In: International Conference on Multimedia and Expo, pp. 209–212 (2002)Google Scholar
  7. 7.
    Bouzaieni, A., Tabbone, S., Barrat, S.: Automatic images annotation extension using a probabilistic graphical model. In: Azzopardi, G., Petkov, N. (eds.) CAIP 2015. LNCS, vol. 9257, pp. 579–590. Springer, Cham (2015). CrossRefGoogle Scholar
  8. 8.
    Wang, F.: A survey on automatic image annotation and trends of the new age. Procedia Eng. 23, 434–438 (2011)CrossRefGoogle Scholar
  9. 9.
    Hanbury, A.: A survey of methods for image annotation. J. Vis. Lang. Comput. 19(5), 617–627 (2008)CrossRefGoogle Scholar
  10. 10.
    Zhang, D., Islam, M.M., Lu, G.: A review on automatic image annotation techniques. Pattern Recogn. 45(1), 346–362 (2012)CrossRefGoogle Scholar
  11. 11.
    Tousch, A.M., Herbin, S., Audibert, J.Y.: Semantic hierarchies for image annotation: a survey. Pattern Recogn. 45(1), 333–345 (2012)CrossRefGoogle Scholar
  12. 12.
    Verma, Y., Jawahar, C.V.: Image annotation using metric learning in semantic neighbourhoods. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7574, pp. 836–849. Springer, Heidelberg (2012). CrossRefGoogle Scholar
  13. 13.
    Neal, R., Hinton, G.: A view of the EM algorithm that justifies incremental, sparse and other variants. In: Jordan, M.I. (ed.) Learning in graphical models, vol. 89, pp. 355–368. Springer, Heidelberg (1998). CrossRefGoogle Scholar
  14. 14.
    Bakliwal, P., Jawahar, C.V.: Active Learning Based Image Annotation. In: NCVPRIPG (2015)Google Scholar
  15. 15.
    Kullback, S., Leibler, R.A.: On information and sufficiency. Ann. Math. Stat. 22(1), 79–86 (1951)MathSciNetCrossRefzbMATHGoogle Scholar
  16. 16.
    Tong, S., Koller, D.: Active learning for parameter estimation in Bayesian networks. NIPS 13, 647–653 (2000)Google Scholar
  17. 17.
    Settles, B.: Active learning literature survey. Univ. Wis. Madison 52(55–66), 11 (2010)Google Scholar
  18. 18.
    Yifan, F., Xingquan, Z., Bin, L.: A survey on instance selection for active learning. Knowl. Inf. Syst. 35(2), 249–283 (2013)CrossRefGoogle Scholar
  19. 19.
    Chengjian, S., Zhu, S., Shi, Z.: Image annotation via deep neural network. In: IAPR ICMVA, pp. 518–521 (2015)Google Scholar
  20. 20.
    Murthy, V.N., Maji, S., Manmatha, R.: Automatic image annotation using deep learning representations. In: ACM ICMR, pp. 603–606 (2015)Google Scholar
  21. 21.
    Karpathy, A., Fei-Fei, L.: Deep visual-semantic alignments for generating image descriptions. CVPR, pp. 3128–3137 (2015)Google Scholar
  22. 22.
    Oliva, A., Torralba, A.: Modeling the shape of the scene: a holistic representation of the spatial envelope. IJCV (2001)Google Scholar
  23. 23.
    Lowe, D.G.: Distinctive image features from scale-invariant keypoints. IJCV 60(2), 91–110 (2004)CrossRefGoogle Scholar
  24. 24.
    Ojala, T., Pietikinen, M., Harwood, D.: A comparative study of texture measures a with classification based on feature distributions. Pattern Recogn. 29, 51–59 (1996)CrossRefGoogle Scholar
  25. 25.
    Bouzaieni, A., Barrat, S., Tabbone, S.: Extension automatique d’annotation d’images en utilisant un modle graphique probabiliste. JFRB (2016)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.LORIA-Université de LorraineVandoeuvre-les-NancyFrance

Personalised recommendations