A Survey of Personalised Image Retrieval and Recommendation

  • Zhenyan Ji
  • Weina Yao
  • Huaiyu Pi
  • Wei Lu
  • Jing He
  • Haishuai Wang
Conference paper
Part of the Communications in Computer and Information Science book series (CCIS, volume 768)


With the advent of web2.0 era, it has been becoming increasingly easy to create and share Internet content. Plenty of pictures are uploaded to the Internet every day. A primary challenge against traditional image retrieval technologies is how to help users quickly discover the images they need. Personalised image retrieval is a new trend in the field of image retrieval. It not only improves the accuracy of the existing retrieval systems, but also better meets the users’ needs. Personalised image retrieval and recommendation (PIRR) can be grouped into two main categories, content-based PIRR and collaborative filtering (CF)-based PIRR. This paper first summarises the development of image retrieval and introduces different image retrieval solutions. Then the key technologies of content-based PIRR are analysed from three aspects, user interest acquisition, user interest representation and personalised implementation. Different techniques are compared and analysed. Regarding CF-based PIRR, the user-based, item-based and model-based CF-based PIRR are introduced and compared. At the end of the paper, we compare and summarise content-based PIRR and CF-based PIRR.


Personalised image retrieval Content-based Collaborative filtering 



This project was supported by NSFC (61272353).


  1. 1.
    Datta, R., Joshi, D., Li, J., Wang, J.Z.: Image retrieval: ideas, influences, and trends of the new age. ACM Comput. Surv. (Csur) 40(2), 5 (2008)CrossRefGoogle Scholar
  2. 2.
    Flickner, M., Sawhney, H., Niblack, W., Ashley, J., Huang, Q., Dom, B., Gorkani, M., Hafner, J., Lee, D., Petkovic, D.: Query by image and video content: the QBIC system. Computer 28(9), 23–32 (1995)CrossRefGoogle Scholar
  3. 3.
    Mehrotra, S., Rui, Y., Chakrabarti, K., Ortega, M., Huang, T.S.: Multimedia analysis and retrieval system. In: Proceedings of the 3rd International Workshop on Information Retrieval Systems (1997)Google Scholar
  4. 4.
    Pentland, A., Picard, R.W., Sclaroff, S.: Photobook: content-based manipulation of image databases. Int. J. Comput. Vision 18(3), 233–254 (1996)CrossRefGoogle Scholar
  5. 5.
    Wilensky, R.: UC Berkeley’s digital library project. Commun. ACM 38(4), 60 (1995)CrossRefGoogle Scholar
  6. 6.
    Smith, J.R., Chang, S.-F.: VisualSEEk: a fully automated content-based image query system. In: Proceedings of the Fourth ACM International Conference on Multimedia, pp. 87–98. ACM (1997)Google Scholar
  7. 7.
    Xu, X., Peng, B., Sun, Z.: A semantic-based image retrieval system: VisEngine. Comput. Eng. 4, 021 (2004)Google Scholar
  8. 8.
    Zhang, H., Wenyin, L., Hu, C.: IFIND—A system for semantics and feature based image retrieval over Internet. In: Proceedings of the Eighth ACM International Conference on Multimedia, pp. 477–478. ACM (2000)Google Scholar
  9. 9.
    Wang, W., Wu, Y., Zhang, A.: SemView: a semantic-sensitive distributed image retrieval system. In: Proceedings of the 2003 Annual National Conference on Digital Government Research, pp. 1–4. Digital Government Society of North America (2003)Google Scholar
  10. 10.
    Ghorab, M.R., Zhou, D., O’Connor, A., Wade, V.: Personalised information retrieval: survey and classification. User Model. User-Adap. Inter. 23(4), 381–443 (2013)CrossRefGoogle Scholar
  11. 11.
    Skowron, M., Tkalčič, M., Ferwerda, B., Schedl, M.: Fusing social media cues: personality prediction from twitter and instagram. In: Proceedings of the 25th International Conference Companion on World Wide Web, pp. 107–108. International World Wide Web Conferences Steering Committee (2016)Google Scholar
  12. 12.
    Liu, D., Hua, X.-S., Wang, M., Zhang, H.: Boost search relevance for tag-based social image retrieval. In: IEEE International Conference on Multimedia and Expo, ICME 2009, pp. 1636–1639. IEEE (2009)Google Scholar
  13. 13.
    Cheung, M., She, J.: Bag-of-features tagging approach for a better recommendation with social big data. In: Proceedings of the 4th International Conference on Advances in Information Mining and Management (IMMM 2014), pp. 83–88 (2014)Google Scholar
  14. 14.
    Sang, J., Xu, C., Lu, D.: Learn to personalized image search from the photo sharing websites. IEEE Trans. Multimedia 14(4), 963–974 (2012)CrossRefGoogle Scholar
  15. 15.
    Qiu, Z.W., Zhang, T.W.: Individuation image retrieval based on user multimedia data management model. Acta Electron. Sin. 36(9), 1746–1749 (2008)Google Scholar
  16. 16.
    Fan, J., Keim, D.A., Gao, Y., Luo, H., Li, Z.: JustClick: personalized image recommendation via exploratory search from large-scale Flickr images. IEEE Trans. Circuits Syst. Video Technol. 19(2), 273–288 (2009)CrossRefGoogle Scholar
  17. 17.
    Yu, J., Tao, D., Wang, M., Rui, Y.: Learning to rank using user clicks and visual features for image retrieval. IEEE Trans. Cybern. 45(4), 767–779 (2015)CrossRefGoogle Scholar
  18. 18.
    Zhang, H., Zha, Z.-J., Yang, Y., Yan, S., Gao, Y., Chua, T.-S.: Attribute-augmented semantic hierarchy: towards bridging semantic gap and intention gap in image retrieval. In: Proceedings of the 21st ACM International Conference on Multimedia, pp. 33–42. ACM (2013)Google Scholar
  19. 19.
    Jayech, K., Mahjoub, M.A.: New approach using Bayesian Network to improve content based image classification systems. IJCSI Int. J. Comput. Sci. Issues 7(6), 53–62 (2010)Google Scholar
  20. 20.
    Hu, T., Yu, J.: Max-margin based Bayesian classifier. Front. Inf. Technol. Electron. Eng. 17(10), 973–981 (2016)CrossRefGoogle Scholar
  21. 21.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097–1105 (2012)Google Scholar
  22. 22.
    Lin, C.-H., Chen, C.-C., Lee, H.-L., Liao, J.-R.: Fast K-means algorithm based on a level histogram for image retrieval. Expert Syst. Appl. 41(7), 3276–3283 (2014)CrossRefGoogle Scholar
  23. 23.
    Liu, Y., Zhang, D., Lu, G., Ma, W.-Y.: A survey of content-based image retrieval with high-level semantics. Pattern Recogn. 40(1), 262–282 (2007)CrossRefMATHGoogle Scholar
  24. 24.
    Wu, J., Xiao, Z.-B., Wang, H.-S., Shen, H.: Learning with both unlabeled data and query logs for image search. Comput. Electr. Eng. 40(3), 964–973 (2014)CrossRefGoogle Scholar
  25. 25.
    Su, J.-H., Huang, W.-J., Philip, S.Y., Tseng, V.S.: Efficient relevance feedback for content-based image retrieval by mining user navigation patterns. IEEE Trans. Knowl. Data Eng. 23(3), 360–372 (2011)CrossRefGoogle Scholar
  26. 26.
    Adomavicius, G., Tuzhilin, A.: Toward the next generation of recommender systems: a survey of the state-of-the-art and possible extensions. IEEE Trans. Knowl. Data Eng. 17(6), 734–749 (2005)CrossRefGoogle Scholar
  27. 27.
    Paik, J.H.: A novel TF-IDF weighting scheme for effective ranking. In: Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 343–352. ACM (2013)Google Scholar
  28. 28.
    Whissell, J.S., Clarke, C.L.: Improving document clustering using Okapi BM25 feature weighting. Inf. Retrieval 14(5), 466–487 (2011)CrossRefGoogle Scholar
  29. 29.
    Zhang, J., Zhuo, L., Shen, L., He, L.: A personalized image retrieval based on user interest model. Int. J. Pattern Recogn. Artif. Intell. 24(03), 401–419 (2010)CrossRefGoogle Scholar
  30. 30.
    Nie, W., Li, X., Liu, A., Su, Y.: 3D object retrieval based on Spatial+LDA model. Multimedia Tools Appl. 76(3), 4091–4104 (2017)CrossRefGoogle Scholar
  31. 31.
    Li, X., Ouyang, J., Lu, Y.: Topic modeling for large-scale text data. Front. Inf. Technol. Electron. Eng. 16(6), 457–465 (2015)CrossRefGoogle Scholar
  32. 32.
    Zhang, Y., Jin, R., Zhou, Z.H.: Understanding bag-of-words model: a statistical framework. Int. J. Mach. Learn. Cybern. 1(1), 43–52 (2010)CrossRefGoogle Scholar
  33. 33.
    Tu, N.A., Dinh, D.-L., Rasel, M.K., Lee, Y.-K.: Topic modeling and improvement of image representation for large-scale image retrieval. Inf. Sci. 366, 99–120 (2016)MathSciNetCrossRefGoogle Scholar
  34. 34.
    Shekhar, R., Jawahar, C.: Word image retrieval using bag of visual words. In: 2012 10th IAPR International Workshop on Document Analysis Systems (DAS), pp. 297–301. IEEE (2012)Google Scholar
  35. 35.
    Yang, J., Jiang, Y.-G., Hauptmann, A.G., Ngo, C.-W.: Evaluating bag-of-visual-words representations in scene classification. In: Proceedings of the International Workshop on Multimedia Information Retrieval, pp. 197–206. ACM (2007)Google Scholar
  36. 36.
    Liu, L.: Contextual topic model based image recommendation system. In: 2015 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT), pp. 239–240. IEEE (2015)Google Scholar
  37. 37.
    Jia, D., Berg, A.C., Li, F.F.: Hierarchical semantic indexing for large scale image retrieval. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 785–792 (2011)Google Scholar
  38. 38.
    Jiang, X., Tan, A.-H.: Learning and inferencing in user ontology for personalized Semantic Web search. Inf. Sci. 179(16), 2794–2808 (2009)CrossRefMATHGoogle Scholar
  39. 39.
    Geng, X., Zhang, H., Bian, J., Chua, T.-S.: Learning image and user features for recommendation in social networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 4274–4282 (2015)Google Scholar
  40. 40.
    Lei, C., Liu, D., Li, W., Zha, Z.J., Li, H.: Comparative deep learning of hybrid representations for image recommendations. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 2545–2553 (2016)Google Scholar
  41. 41.
    Song, G., Jin, X., Chen, G., Nie, Y.: Two-level hierarchical feature learning for image classification. Front. Inf. Technol. Electron. Eng. 17(9), 897–906 (2016)CrossRefGoogle Scholar
  42. 42.
    Burdescu, D.D., Mihai, C.G., Stanescu, L., Brezovan, M.: Automatic image annotation and semantic based image retrieval for medical domain. Neurocomputing 109, 33–48 (2013)CrossRefGoogle Scholar
  43. 43.
    Kurtz, C., Depeursinge, A., Napel, S., Beaulieu, C.F., Rubin, D.L.: On combining image-based and ontological semantic dissimilarities for medical image retrieval applications. Med. Image Anal. 18(7), 1082–1100 (2014)CrossRefGoogle Scholar
  44. 44.
    Kurtz, C., Beaulieu, C.F., Napel, S., Rubin, D.L.: A hierarchical knowledge-based approach for retrieving similar medical images described with semantic annotations. J. Biomed. Inform. 49, 227–244 (2014)CrossRefGoogle Scholar
  45. 45.
    Rui, Y., Huang, T.S.: A novel relevance feedback technique in image retrieval. In: Proceedings of the Seventh ACM International Conference on Multimedia (Part 2), pp. 67–70. ACM (1999)Google Scholar
  46. 46.
    Rocchio, J.J.: Relevance feedback in information retrieval. In: Salton, G. (ed.) The Smart Retrieval System: Experiments in Automatic Document Processing, pp. 313–323. Prentice Hall Inc., Englewood Cliffs (1971)Google Scholar
  47. 47.
    Porkaew, K., Chakrabarti, K.: Query refinement for multimedia similarity retrieval in MARS. In: Proceedings of the Seventh ACM International Conference on Multimedia (Part 1), pp. 235–238. ACM (1999)Google Scholar
  48. 48.
    Tao, D., Tang, X., Li, X., Wu, X.: Asymmetric bagging and random subspace for support vector machines-based relevance feedback in image retrieval. IEEE Trans. Pattern Anal. Mach. Intell. 28(7), 1088–1099 (2006)CrossRefGoogle Scholar
  49. 49.
    Johnson, M., Shotton, J., Cipolla, R.: Semantic texton forests for image categorization and segmentation. In: Criminisi, A., Shotton, J. (eds.) Decision Forests for Computer Vision and Medical Image Analysis. Advances in Computer Vision and Pattern Recognition, pp. 211–227. Springer, London (2013)CrossRefGoogle Scholar
  50. 50.
    Tong, S., Chang, E.: Support vector machine active learning for image retrieval. In: Proceedings of the Ninth ACM International Conference on Multimedia, pp. 107–118. ACM (2001)Google Scholar
  51. 51.
    Zhao, S., Du, N., Nauerz, A., Zhang, X., Yuan, Q., Fu, R.: Improved recommendation based on collaborative tagging behaviors. In: Proceedings of the 13th International Conference on Intelligent User Interfaces, pp. 413–416. ACM (2008)Google Scholar
  52. 52.
    Gong, S.J.: Personalized recommendation system based on association rules mining and collaborative filtering. In: Wang, Y. (ed.) Applied Mechanics and Materials, pp. 540–544. Trans Tech Publ, Zürich (2011)Google Scholar
  53. 53.
    Ju, B., Qian, Y., Ye, M.: Preference transfer model in collaborative filtering for implicit data. Front. Inf. Technol. Electron. Eng. 17(6), 489–500 (2016)CrossRefGoogle Scholar
  54. 54.
    Sarwar, B., Karypis, G., Konstan, J., Riedl, J.: Item-based collaborative filtering recommendation algorithms. In: Proceedings of the 10th International Conference on World Wide Web, pp. 285–295. ACM (2001)Google Scholar
  55. 55.
    Zhou, K., Yang, S.-H., Zha, H.: Functional matrix factorizations for cold-start recommendation. In: Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 315–324. ACM (2011)Google Scholar
  56. 56.
    Yuan, Z., Huang, C., Sun, X., Li, X., Xu, D.: A microblog recommendation algorithm based on social tagging and a temporal interest evolution model. Front. Inf. Technol. Electron. Eng. 16(7), 532–540 (2015)CrossRefGoogle Scholar
  57. 57.
    Tiraweerakhajohn, C., Pinngern, O.: A combination of content-based filtering and item-based collaborative filtering using association rules. In: The 1st International Conference on Electrical Engineering/Electronics, Computer, Telecommunications and Information Technology. ECTI, Thailand (2004)Google Scholar
  58. 58.
    Lei, W., Qing, F., Zhou, J.: Improved personalized recommendation based on causal association rule and collaborative filtering. Int. J. Distance Educ. Technol. (IJDET) 14(3), 21–33 (2016)CrossRefGoogle Scholar
  59. 59.
    Ye, H.: A personalized collaborative filtering recommendation using association rules mining and self-organizing map. JSW 6(4), 732–739 (2011)CrossRefGoogle Scholar
  60. 60.
    Thorat, P.B., Goudar, R., Barve, S.: Survey on collaborative filtering, content-based filtering and hybrid recommendation system. Int. J. Comput. Appl. 110(4), 31–36 (2015)Google Scholar
  61. 61.
    Ma, Z., Leijon, A.: A model-based collaborative filtering method for bounded support data. In: 2012 3rd IEEE International Conference on Network Infrastructure and Digital Content (IC-NIDC), pp. 545–548. IEEE (2012)Google Scholar
  62. 62.
    Fernández-Tobías, I., Braunhofer, M., Elahi, M., Ricci, F., Cantador, I.: Alleviating the new user problem in collaborative filtering by exploiting personality information. User Model. User-Adap. Inter. 26(2–3), 221–255 (2016)CrossRefGoogle Scholar
  63. 63.
    Yang, C., Zhou, Y., Chen, L., Zhang, X., Chiu, D.M.: Social-group-based ranking algorithms for cold-start video recommendation. Int. J. Data Sci. Anal. 1(3–4), 165–175 (2016)CrossRefGoogle Scholar
  64. 64.
    Candillier, L., Meyer, F., Boullé, M.: Comparing state-of-the-art collaborative filtering systems. In: Perner, P. (ed.) MLDM 2007. LNCS, vol. 4571, pp. 548–562. Springer, Heidelberg (2007). doi: 10.1007/978-3-540-73499-4_41 CrossRefGoogle Scholar
  65. 65.
    Sanchez, F., Barrilero, M., Uribe, S., Alvarez, F., Tena, A., Menendez, J.M.: Social and content hybrid image recommender system for mobile social networks. Mob. Netw. Appl. 17(6), 782–795 (2012)CrossRefGoogle Scholar
  66. 66.
    Lekakos, G., Caravelas, P.: A hybrid approach for movie recommendation. Multimedia Tools Appl. 36(1), 55–70 (2008)CrossRefGoogle Scholar
  67. 67.
    Widisinghe, A., Ranasinghe, D., Kulathilaka, K., Kaluarachchi, R., Wimalawarne, K.A.D.N.K.: picSEEK: collaborative filtering for context-based image recommendation. In: International Conference on Information and Automation for Sustainability, pp. 225–232 (2010)Google Scholar
  68. 68.
    Liu, X., Tsai, M.H., Huang, T.: Analyzing user preference for social image recommendation. arXiv:1604.07044 [cs.IR] (2016)

Copyright information

© Springer Nature Singapore Pte Ltd. 2017

Authors and Affiliations

  • Zhenyan Ji
    • 1
  • Weina Yao
    • 1
  • Huaiyu Pi
    • 1
  • Wei Lu
    • 1
  • Jing He
    • 2
  • Haishuai Wang
    • 3
  1. 1.Beijing Jiaotong UniversityBeijingChina
  2. 2.Victoria UniversityMelbourneAustralia
  3. 3.Washington University in St. LouisSt. LouisUSA

Personalised recommendations