International Journal of Computer Vision

, Volume 56, Issue 1–2, pp 65–77 | Cite as

Learning from User Behavior in Image Retrieval: Application of Market Basket Analysis

  • Henning Müller
  • Thierry Pun
  • David Squire


This article describes an approach to learn feature weights for content-based image retrieval (CBIR) from user interaction log files. These usage log files are analyzed for images marked together by a user in the same query step. The problem is somewhat similar to one of the traditional data mining problems, the market basket analysis problem, where items bought together in a supermarket are analyzed. This paper outlines similarities and differences between the two fields and explains how to use the interaction data for deriving a better feature weighting.

Experiments with existing log files are done and a significant improvement in performance is reached with a feature weighting calculated from the information contained in the log files. Even with several steps of relevance feedback the results remain much better than without the learning, which means that not only information from feedback is taken into account earlier, but a better quality of retrieval is reached in all steps.

content-based image retrieval market basket analysis learning from user interaction 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Agrawal, R., Imielinski, T., and Swami, A. 1993. Mining association rules between sets of items in large databases. In Proceedings of the 1993 ACM SIGMOD Conference, Washington DC, USA, pp. 207–216.Google Scholar
  2. Agrawal, R. and Srikant, R. 1994. Fast algorithms for mining association rules. In Proceedings of the 20th VLDB Conference Santiago, Chile, pp. 487–499.Google Scholar
  3. Annotated groundtruth database. 1999. Department of Computer Science and Engineering, University of Washington, http:// groundtruth/.Google Scholar
  4. Berendt, B. and Spiliopoulou, M. 2000. Analysis of navigation behaviour in web sites integrating multiple information systems. VLDB Journal: Special Issue on Databases and theWeb, 9(1):56–75.Google Scholar
  5. Brin, S., Motwani, R., and Silverstein, C. 1997. Beyond market baskets: Generalizing association rules to correlations. In Proceedings of the Annual International ACM SIGMOD Conference on Research and Development in Management of Data (SIGMOD'97), J. Peckham (Ed.), Tuscon AR, USA, pp. 255–264.Google Scholar
  6. Cox, I.J., Miller, M.L., Omohundro, S.M., and Yianilos, P.N. 1996. Target testing and the PicHunter Bayesian multimedia retrieval system. In Advances in Digital Libraries (ADL'96), Library of Congress: Washington, DC, pp. 66–75.Google Scholar
  7. Han, J., Pei, J., and Yin, Y. 2000. Mining frequent patterns without candicate generation. In Proceedings of the Annual International ACMSIGMODConference on Research andDevelopment in Management of Data (SIGMOD2000), W. Chen, J.F. Naughton, and P.A. Bernstein (Eds.), Dallas, TX, USA.Google Scholar
  8. Harman, D. 1992. Overview of the first Text REtrieval Conference (TREC–1). In Proceedings of the First Text REtrieval Conference (TREC–1), Washington DC, USA, pp. 1–20.Google Scholar
  9. Hipp, J., Gützer, U., and Nakhaeizadeh, G. 2000. Algorithms for association rule mining—A general survey and comparison. SIGKDD Explorations, 2(1):58–64.Google Scholar
  10. Jermain, C. and Miller, R.J. 2001. Association mining without support thresholds. Technical report, Georgia Institute of TechnologyGoogle Scholar
  11. Kohrs, A. and Merialdo, B. 1999. Clustering for collaborative filtering applications. In Proceedings of the International Conference on Computational Intelligence for Modelling Control andAutomation. IOS Press: Vienna, Austria, pp. 199–204.Google Scholar
  12. Lee, C.S., Ma, W.-Y., and Zhang, H. 1999. Information embedding based on user's relevance feedback in image retrieval. In Multimedia Storage and Archiving Systems IV(VV03) vol. 3246 of SPIE Proceedings, S. Panchanathan, S.-F. Chang and C.-C.J. Kuo (Eds.), pp. 294–304 (SPIE Symposium on Voice, Video and Data Communications).Google Scholar
  13. Li, B., Chang, E., and Li, C.-S. 2001. Learning image query concepts via intelligent sampling. In Proceedings of the Second International Conference on Multimedia and Exposition (ICME'2001), IEEE Computer Society: Tokyo, Japan, pp. 1168–1171.Google Scholar
  14. Li, M., Chen, Z., Wenyin, L., and Zhang, H.-J. 2001. A statistical correlation model for image retrieval. In Proceedings of the ACM MultimediaWorkshop on Multimedia Information Retrieval (ACM MIR 2001), The Association for Computing Machinery: Ottawa, Canada, pp. 42–45.Google Scholar
  15. Ma, W.Y., Deng, Y., and Manjunath, B.S. 1997. Tools for texture and color-based search of images. In Human Vision and Electronic Imaging II, volume 3016 of SPIE Proceedings, B.E. Rogowitz and T.N. Pappas (Eds.), San Jose, CA, pp. 496–507.Google Scholar
  16. Mannila, H. and Toivonen, H. 1996. Discovering generalized episodes using minimal occurences. In Proceedings of the 2nd International Conference on Knowledge Discovery and Data Mining (SIGKDD1996), Portland, OR, USA, pp. 146–151.Google Scholar
  17. Minka, T. 1996. An image database browser that learns from user interaction. Master's thesis, MIT Media Laboratory, 20 Ames St., Cambridge, MA 02139.Google Scholar
  18. Müller, H., Müller, W., Squire, D.M., Marchand-Maillet, S., and Pun, T. 2000a. Learning feature weights from user behavior in contentbased image retrieval. In ACMSIGKDD International Conference on Knowledge Discovery and Data Mining (Workshop on Multimedia Data Mining MDM/KDD2000), S. Simoff and O. Zaiane (Eds.). Boston, MA, USA, pp. 67–72.Google Scholar
  19. Müller, H., Müller, W., Squire, D.M., Marchand-Maillet, S., and Pun, T. 2000b. Strategies for positive and negative relevance feedback in image retrieval. In Proceedings of the 15th International Conference on Pattern Recognition (ICPR 2000), A. Sanfeliu, J.J. Villanueva, M. Vanrell, R. Alcézar, J.-O. Eklundh, and Y. Aloimonos (Eds.) IEEE: Barcelona, Spain, pp. 1043–1046.Google Scholar
  20. Müller, H., Müller, W., Squire, D.M., Marchand-Maillet, S., and Pun, T. 2001. Performance evaluation in content-based image retrieval: Overview and proposals. Pattern Recognition Letters, 22(5):593–601.Google Scholar
  21. Müller, W., Pečenović, Z., Müller, H., Marchand-Maillet, S., Pun, T., Squire, D.M., Vries, A.P.D., and Giess, C. 2000c. MRML: An extensible communication protocol for interoperability and benchmarking of multimedia information retrieval systems. In SPIE Photonics East—Voice, Video, and Data Communications, Boston, MA, USA, pp. 961–968.Google Scholar
  22. Müller, W., Squire, D.M., Müller, H., and Pun, T. 1999. Hunting moving targets: An extension to Byesian methods in multimedia databases. In Multimedia Storage and Archiving Systems IV(VV02), vol. 3846 of SPIE Proceedings, S. Panchanathan, S.-F. Chang, and C.-C.J. Kuo (Eds.), pp. 328–337 (SPIE Symposium on Voice, Video and Data Communications).Google Scholar
  23. Panchanathan, S., Chang, S.-F., and Kuo, C.-C.J. (Eds.). 1999. Multimedia Storage and Archiving Systems IV (VV02), vol. 3846 of SPIE Proceedings. Boston, Massachusetts, USA. (SPIE Symposium on Voice, Video and Data Communications).Google Scholar
  24. Rui, Y., Huang, T.S., Ortega, M., and Mehrotra, S. 1998. Relevance feedback: A power tool for interactive content-based image retrieval. IEEE Transactions on Circuits and Systems for Video Technology, 8(5):644–655. (Special Issue on Segmentation, Description, and Retrieval of Video Content).Google Scholar
  25. Salton, G. and Buckley, C. 1988. Term weighting approaches in automatic text retrieval. Information Processing and Management, 24(5):513–523.Google Scholar
  26. Salton, G. and Buckley, C. 1990. Improving retrieval performance by relevance feedback. Journal of the American Society for Information Science, 41(4):288–287.Google Scholar
  27. Savasere, A., Omiecinski, E., and Navathe, S. 1995. An efficient algorithm for mining association rules in large databases. In Proceedings of the 22nd International Conference on Very Large Databases (VLDB'95), U. Dayal, P.M.D. Gray, and S. Nishio (Eds.), Zürich, Switzerland.Google Scholar
  28. Shekhar, S. and Huang, Y. 2001. Discovering spatial co-location patterns: A summary of results. In Proceedings of the 7th International Symposium on Spatial and Temporal Databases, Retondo Beach CA, USA.Google Scholar
  29. Smeulders, A.W.M., Worring, M., Santini, S., Gupta, A., and Jain, R. 2000. Content-based image retrieval at the end of the early years. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(12):1349–1380.Google Scholar
  30. Smith, J.R. 1998. Image retrieval evaluation. In IEEE Workshop on Content-Based Access of Image andVideo Libraries (CBAIVL'98). Santa Barbara, CA, USA, pp. 112–113.Google Scholar
  31. Smith, J.R. and Chang, S.-F. 1996. VisualSEEk: A fully automated content-based image query system. In The Fourth ACM International Multimedia Conference and Exhibition. Boston, MA, USA.Google Scholar
  32. Squire, D.M., Müller, W., Müller, H., and Pun, T. 2000. Content-based query of image databases: Inspirations from text retrieval. In Pattern Recognition Letters (Selected Papers from The 11th Scandinavian Conference on Image Analysis SCIA' 99), B.K. Ersboll and P. Johansen (Eds.), 21(13/14):1193–1198.Google Scholar
  33. Vasconcelos, N. and Lippman, A. 2000. Learning over multiple temporal scales in image databases. In 6th European Conference on Computer Vision (ECCV2000), D. Vernon (Ed.), number 1842 in Lecture Notes in Computer Science. Springer-Verlag: Dublin, Ireland, pp. 33–47.Google Scholar
  34. Voorhees, E.M. and Harmann, D. 1998. Overview of the seventh Text REtrieval Conference (TREC–7). In The Seventh Text Retrieval Conference. Gaithersburg, MD, USA, pp.1–23.Google Scholar
  35. Worring, M., Smeulders, A.W.M., and Santini, S. 2000. Interaction in content-based image retrieval: An evaluation of the state of the art. In Fourth International Conference on Visual Information Systems (VISUAL'2000), R. Laurini (Ed.), number 1929 in Lecture Notes in Computer Science, Springer–Verlag: Lyon, France, pp. 26–36.Google Scholar
  36. Wu, K.-L., Yu, P.S., and Ballman, A. 1998. Speedtracer: A web usage mining and analysis tool. IBM Systems Journal on Internet Computing, 37(1):89–105.Google Scholar
  37. Zaki, M.J., Parthgasarathy, S., Ogihara, M., and Li, W. 1997. New algorithms for fast discovery of association rules. In Proceedings of the 3rd International Conference on Knowledge Discovery and Data Mining (SIGKDD1997). Newport Beach, CA, USA.Google Scholar

Copyright information

© Kluwer Academic Publishers 2004

Authors and Affiliations

  • Henning Müller
    • 1
  • Thierry Pun
    • 1
  • David Squire
    • 2
  1. 1.Computer Vision GroupUniversity of GenevaGenève 4Switzerland
  2. 2.Computer Science and Software EngineeringMonash UniversityMelbourneAustralia

Personalised recommendations