Multimedia Tools and Applications

, Volume 64, Issue 3, pp 695–716 | Cite as

The Mosaic Test: measuring the effectiveness of colour-based image retrieval

  • William PlantEmail author
  • Joanna Lumsden
  • Ian T. Nabney


A variety of content-based image retrieval systems exist which enable users to perform image retrieval based on colour content—i.e., colour-based image retrieval. For the production of media for use in television and film, colour-based image retrieval is useful for retrieving specifically coloured animations, graphics or videos from large databases (by comparing user queries to the colour content of extracted key frames). It is also useful to graphic artists creating realistic computer-generated imagery (CGI). Unfortunately, current methods for evaluating colour-based image retrieval systems have 2 major drawbacks. Firstly, the relevance of images retrieved during the task cannot be measured reliably. Secondly, existing methods do not account for the creative design activity known as reflection-in-action. Consequently, the development and application of novel and potentially more effective colour-based image retrieval approaches, better supporting the large number of users creating media for use in television and film productions, is not possible as their efficacy cannot be reliably measured and compared to existing technologies. As a solution to the problem, this paper introduces the Mosaic Test. The Mosaic Test is a user-based evaluation approach in which participants complete an image mosaic of a predetermined target image, using the colour-based image retrieval system that is being evaluated. In this paper, we introduce the Mosaic Test and report on a user evaluation. The findings of the study reveal that the Mosaic Test overcomes the 2 major drawbacks associated with existing evaluation methods and does not require expert participants.


Image retrieval Image databases Content-based image retrieval Query-by-sketch Query-by-colour Performance evaluation 


  1. 1.
    Datta R, Joshi D, Li J, Wang JZ (2008) Image retrieval: ideas, influences, and trends of the new age. ACM Comput Surv 40(2):1–60CrossRefGoogle Scholar
  2. 2.
    Faloutsos C, Equitz W, Flickner M, Niblack W, Petkovic D, Barber R (1994) Efficient and effective querying by image content. J Intell Inf Syst 3:231–262CrossRefGoogle Scholar
  3. 3.
    Google (2010) Google images. Accessed 2 Nov 2010
  4. 4.
    Hart SG (2006) NASA-Task load index (NASA-TLX); 20 years later. In: Proceedings of the human factors and ergonomics society 50th annual meeting, pp 904–908Google Scholar
  5. 5.
    Huang J, Kumar SR, Mitra M, Zhu W, Zabih R (1997) Image indexing using color correlograms. In: Computer vision and pattern recognition, pp 762–768Google Scholar
  6. 6.
    Huiskes MJ, Lew MS (2008) The MIR flickr retrieval evaluation. In: ACM international conference on multimedia information retrieval, pp 39–43Google Scholar
  7. 7. (2010) Accessed 1 Dec 2010
  8. 8.
    MacDonald L (1999). Using color effectively in computer graphics. IEEE Comput Graph Appl 19(4):20–35CrossRefGoogle Scholar
  9. 9.
    Microsoft (2011) Bing images. Accessed 27 Jan 2011
  10. 10.
    Nakade S, Karule P (2007) Mosaicture: image mosaic generating system using CBIR technique. In: International conference on computational intelligence and multimedia applications, pp 339–343Google Scholar
  11. 11.
    Ortega M, Rui Y, Chakrabarti K, Mehrotra S, Huang TS (1997) Supporting similarity queries in MARS. In: ACM international multimedia conference, pp 403–413Google Scholar
  12. 12.
    Plant W, Schaefer G (2009) Evaluation and benchmarking of image database navigation tools. In: International conference on image processing, computer vision, and pattern recognition, pp 248–254Google Scholar
  13. 13.
    Rodden K, Wood K (2003) How do people manage their digital photographs? In: SIGCHI conference on human factors in computing systems, pp 409–416Google Scholar
  14. 14.
    Rodden K, Basalaj W, Sinclair D, Wood K (2001) Does organisation by similarity assist image browsing? In: SIGCHI conference on human factors in computing systems, pp 190–197Google Scholar
  15. 15.
    Schaefer G, Stich M (2004) UCID—an uncompressed colour image database. In: Storage and retrieval methods and applications for multimedia, pp 472–480Google Scholar
  16. 16.
    Schön DA (1983) The reflective practitioner: how professionals think in action. Basic BooksGoogle Scholar
  17. 17.
    Sikora T (2001) The MPEG-7 visual standard for content description—an overview. IEEE Trans. Circuits Syst Video Technol 11(6):696–702MathSciNetCrossRefGoogle Scholar
  18. 18.
    Silvers R (1996) Photomosaics: putting pictures in their place. Master’s thesis, Massachusetts Institute of TechnologyGoogle Scholar
  19. 19.
    Swain M, Ballard D (1991) Color indexing. Int J Comput Vis 7(1):11–32CrossRefGoogle Scholar
  20. 20.
    Terry M, Mynatt ED (2002) Recognizing creative needs in user interface design. In: Creativity and cognition, pp 38–44Google Scholar
  21. 21.
    Truong BT, Venkates S (2007) Video abstraction: a systematic review and classification. ACM Trans Multimedia Comput Commun Appl 3(1):1–37CrossRefGoogle Scholar
  22. 22.
    Vijfwinkel M (2009) CG textures. Accessed Oct 2009
  23. 23.
    Yahoo (2009) Flickr. Accessed Oct 2009
  24. 24.
    Zhang Y, Nascimento M, Zaiane O (2003) Building image mosaics: an application of content-based image retrieval. In: International conference on multimedia and expo, pp 317–320Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2012

Authors and Affiliations

  1. 1.Computer Science, School of Engineering and Applied ScienceAston UniversityBirminghamUK

Personalised recommendations