Multimedia Tools and Applications

, Volume 54, Issue 2, pp 473–497 | Cite as

Task-based annotation and retrieval for image information management

  • Dympna O’SullivanEmail author
  • David C. Wilson
  • Michela Bertolotto


Continuing advances in digital image capture and storage are resulting in a proliferation of imagery and associated problems of information overload in image domains. In this work we present a framework that supports image management using an interactive approach that captures and reuses task-based contextual information. Our framework models the relationship between images and domain tasks they support by monitoring the interactive manipulation and annotation of task-relevant imagery. During image analysis, interactions are captured and a task context is dynamically constructed so that human expertise, proficiency and knowledge can be leveraged to support other users in carrying out similar domain tasks using case-based reasoning techniques. In this article we present our framework for capturing task context and describe how we have implemented the framework as two image retrieval applications in the geo-spatial and medical domains. We present an evaluation that tests the efficiency of our algorithms for retrieving image context information and the effectiveness of the framework for carrying out goal-directed image tasks.


Task-based information retrieval Capturing and reusing user context Image manipulation Semantic annotation Case-based reasoning 



The support of the Proof of Concept and Commercialization Funds of Enterprise Ireland is gratefully acknowledged.


  1. 1.
    Allen J (2002) Challenges in information retrieval and language modelling: report of a workshop held at the centre for intelligent information retrieval, University of Massachusetts. SIGIR Forum 37:31–47CrossRefGoogle Scholar
  2. 2.
    Althoff K, Webber R (2006) Knowledge management in case-based reasoning. Knowledge Eng Rev 20:305–310CrossRefGoogle Scholar
  3. 3.
    Andriole KP, Morin RL, Arenson RL, Carrino JA, Erickson BJ, Horii SC, Piraino DW, Reiner BI, Seibert JA, Siegel E (2004) Addressing the coming radiology crisis: the society for computer applications in radiology transforming the radiological interpretation process (trip) initiative. J Digit Imaging 17:235–243CrossRefGoogle Scholar
  4. 4.
    Belkin NJ, Callan J (2003) Context-based information access. Report of the Discussion Group on Context-Based Information Access of the Workshop on Information Retrieval and Databases: Synergies and Syntheses National Science Foundation.
  5. 5.
    Bichindaritz I (2003) Solving safety implications in a case-based decision-support system in medicine. Proceedings of the Fifth International Conference on Case-based Reasoning Workshop on Case-Based Reasoning in the Health Sciences, pp 178–183Google Scholar
  6. 6.
    Bichindaritz I (2008) Prototypical case mining from biomedical literature for bootstrapping a case base. Appl Intell 28:222–237CrossRefGoogle Scholar
  7. 7.
    Bichindaritz I, Marling C (2006) Case-based reasoning in the health sciences: what’s next? Artif Intell Med 36:127–135CrossRefGoogle Scholar
  8. 8.
    Bradley F, Jung B (2005) Putting fun into function with QuizMed—an interactive medical application. Proceedings of the Eighteenth International Conference on Computer Based Medical Systems, pp 226–231Google Scholar
  9. 9.
    Budzik J, Hammond KJ (2000) User interactions with everyday applications as context for just-in-time information access. Proceedings of the Fifth International Conference on Intelligent User Interfaces pp 44–51Google Scholar
  10. 10.
    Budzik J, McLoughlin L, Hammond K (2003) Information access in context: experiences with the Watson system. Dissertation, Northwestern UniversityGoogle Scholar
  11. 11.
    Burke R, Kass A (2000) Retrieving stories for case-based teaching. In: Leake D (ed) Case-based reasoning: experiences, lessons, and future directions, 2nd edn. AAAI/MIT, pp 93–109Google Scholar
  12. 12.
    Claypool M, Le P, Waseda M, Brown D (2001) Implicit interest indicators. Proceedings of the Sixth International Conference on Intelligent User Interfaces, pp 33–40Google Scholar
  13. 13.
    Demner-Fushman D, Antani S, Simpson M, Thoma G (2009) Annotation and retrieval of clinically relevant images. Int J MedInform 78:59–67Google Scholar
  14. 14.
    DermAtlas: Online Dermatology Image Library (2009) Accessed 10 Apr. Available from
  15. 15.
    Enser PGB, Sandom CJ, Lewis PH (2005) Automatic annotation of images from the practitioner perspective. Proceedings of the 4th International Conference on Image and Video Retrieval, pp 497–506Google Scholar
  16. 16.
    Fan J, Gao Y, Luo H (2004) Multi-level annotation of natural scenes using dominant image components and semantic concepts. Proceedings of the ACM International Conference on Multimedia, pp 540–547Google Scholar
  17. 17.
    Flickner M, Sawhney H, Ashley J, Huang Q, Dom B, Gorkani M, Hafner J, Lee D, Petkovic D, Steele D, Yanker P (1995) Query by image and video content: the qbic system. IEEE Comput 28:23–32Google Scholar
  18. 18.
    Frucci M, Perner P, di Baja GS (2008) Case-based-reasoning for image segmentation. Int J Pattern Recognit Artif Intell 22:829–842CrossRefGoogle Scholar
  19. 19.
    Gandhi V, Kang JM, Shekhar S (2009) In: Spatial databases: encyclopaedia of computer science and engineering. Wiley, New YorkGoogle Scholar
  20. 20.
    Grimnes M, Aamodt A (1996) A two layer case-based reasoning architecture for medical image understanding. Proceedings of the Second European Workshop on Case-based Reasoning, pp 164–178Google Scholar
  21. 21.
    Hare JS, Lewis PH, Enser PGB, Sandom CJ (2006) Mind the gap: another look at the problem of the semantic gap in image retrieval. In: Chang EY, Hanjalic A, Sebe N (eds) Multimedia Content Analysis, Management, and Retrieval 6073:1–12Google Scholar
  22. 22.
    Holt A, Bichindaritz I, Schmidt R, Perner P (2006) Medical applications in case-based reasoning. Knowledge Eng Rev 20:289–292CrossRefGoogle Scholar
  23. 23.
    Hussain F, Abidi SSR (2005) A knowledge management framework to morph clinical cases with clinical practice guidelines. Stud Health Technol Inform 116:731–736Google Scholar
  24. 24.
    Kelly D, Teevan J (2003) Implicit feedback for inferring user preference: a bibliography. SIGIR Forum 37:18–28CrossRefzbMATHGoogle Scholar
  25. 25.
    Lewis L, Foxx L (2005) NASA Takes Google on Journey into Space. Accessed 6 Jul 2009. Available from
  26. 26.
    Lieberman H (1995) Letizia: An agent that assists web browsing. Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence, pp 924–929Google Scholar
  27. 27.
    Marling C, Whitehouse P (2001) Case-based reasoning in the care of Alzheimer’s disease patients. Proceedings of the Fourth International Conference on Case-Based Reasoning, pp 702–715Google Scholar
  28. 28.
    Minor M (2006) Experience management with case-based assistant systems. Proceedings of the Eighth European Conference on Case-Based Reasoning, pp 182–195Google Scholar
  29. 29.
    Müller H, Kalpathy-Cramer J, Kahn CE Jr, Hatt W, Bedrick S, Hersh W (2008) Overview of the ImageCLEFmed 2008 medical image retrieval task. Available from:
  30. 30.
    Nilsson M, Sollenborn M (2004) Advancements and trends in medical case-based reasoning: An overview of systems and system development. Proceedings of the Seventeenth International Florida Artificial Intelligence Research Society Conference, pp 178–183Google Scholar
  31. 31.
    O’Sullivan D, Smyth B, Wilson D (2003) Explicit vs. implicit profiling: a case-study in electronic programme guides. Proceedings of the Eighteenth International Joint Conference on Artificial Intelligence, pp 1351–1359Google Scholar
  32. 32.
    Perner P, Holt A, Richter M (2006) Image processing in case-based reasoning. Knowledge Eng Rev 20:311–314CrossRefGoogle Scholar
  33. 33.
    Rhodes B (2003) Using physical context for just-in-time information retrieval. IEEE Trans Comput 52:1011–1014CrossRefGoogle Scholar
  34. 34.
    Salton G, McGill M (1983) Introduction to modern information retrieval. McGraw-Hill.Google Scholar
  35. 35.
    Solomon P (2003) Looking for information—a survey of research on information seeking, needs, and behaviour. Inf Retrieval 6:284–288CrossRefGoogle Scholar
  36. 36.
    The Apache Lucene (2008) The Apache Software Foundation; Accessed 30 Oct. Available from
  37. 37.
    Uchihashi S, Kanade T (2005) Content-free image retrieval based on relations exploited from user feedbacks. Proceedings of IEEE International Conference on Multimedia and Expo, pp 1358–1361Google Scholar
  38. 38.
    Wang J, Li J (2003) Automatic linguistic indexing of pictures by a statistical modelling approach. IEEE T Pattern Ana 25:1075–1088CrossRefGoogle Scholar
  39. 39.
    Weber R, Aha DW, Branting K, Lucas JR, Becerra-Fernandez I (2000) Active case-based reasoning for lessons delivery system. Proceedings of the Thirtieth Florida Artificial Intelligence Research Society Conference, pp 170–174Google Scholar
  40. 40.
    Worring M, Schrieber G (2007) Semantic image and video indexing in broad domains. IEEE Trans Multimedia 9:909–919CrossRefGoogle Scholar
  41. 41.
    Zhao R, Grosky WI (2000) Negotiating the semantic gap: from feature maps to semantic landscapes. Pattern Recogn 35:593–600CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2010

Authors and Affiliations

  • Dympna O’Sullivan
    • 1
    Email author
  • David C. Wilson
    • 2
  • Michela Bertolotto
    • 3
  1. 1.School of Engineering and Applied ScienceUniversity of AstonBirminghamUK
  2. 2.Department of Software and Information SystemsUniversity of North CarolinaCharlotteUSA
  3. 3.School of Computer Science and InformaticsUniversity College DublinDublinIreland

Personalised recommendations