An Interactive Paper and Digital Pen Interface for Query-by-Sketch Image Retrieval

  • Roman Kreuzer
  • Michael Springmann
  • Ihab Al Kabary
  • Heiko Schuldt
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7224)


A major challenge when dealing with large collections of digital images is to find relevant objects, especially when no metadata on the objects is available. Content-based image retrieval (CBIR) addresses this problem but usually lacks query images that are good enough to express the user’s information need. Therefore, in Query-by-Sketch, CBIR has been considered with user provided sketches as query objects – but so far, this has suffered from the limitations of existing user interfaces. In this paper, we present a novel user interface for query by sketch that exploits emergent interactive paper and digital pen technology. Users can draw sketches on paper in a user-friendly way. Search can be started interactively from the paper front-end, due to a streaming interface from the digital pen to the underlying CBIR system. We present the implementation of the interactive paper/digital pen interface on top of QbS, our system for CBIR using sketches, and we present in detail the evaluation of the system on the basis of the MIRFLICKR-25000 image collection.


Image Retrieval Target Image Graphic Tablet Interactive Paper Edge Histogram Descriptor 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Agosti, M., Berretti, S., Brettlecker, G., del Bimbo, A., Ferro, N., Fuhr, N., Keim, D.A., Klas, C.-P., Lidy, T., Milano, D., Norrie, M.C., Ranaldi, P., Rauber, A., Schek, H.-J., Schreck, T., Schuldt, H., Signer, B., Springmann, M.: DelosDLMS: The Integrated DELOS Digital Library Management System. In: Digital Libraries: Research and Development – 1st Int. DELOS Conf., pp. 36–45 (February 2007)Google Scholar
  2. 2.
    Chalechale, A., Mertins, A., Naghdy, G.: Edge Image Description using Angular Radial Partitioning. IEE Vision, Image & Signal Processing 151(2) (2004)Google Scholar
  3. 3.
    Chen, T., Cheng, M.-M., Tan, P., Shamir, A., Hu, S.-M.: Sketch2photo: internet image montage. ACM Trans. Graph. 28(5) (2009)Google Scholar
  4. 4.
    del Bimbo, A., Pala, P.: Visual Image Retrieval by Elastic Matching of User Sketches. IEEE Trans. on Pattern Analysis & Machine Intelligence 19 (1997)Google Scholar
  5. 5.
    Eitz, M., Hildebrand, K., Boubekeur, T., Alexa, M.: PhotoSketch: A Sketch based Image Query and Compositing System. In: ACM SIGGRAPH (August 2009)Google Scholar
  6. 6.
    Flickner, M., Sawhney, H., Niblack, W., Ashley, J., Huang, Q., Dom, B., Gorkani, M., Hafner, J., Lee, D., Petkovic, D., Steele, D., Yanker, P.: Query by Image and Video Content: The QBIC System. Computer 28(9), 23–32 (1995)CrossRefGoogle Scholar
  7. 7.
    Hirata, K., Kato, T.: Query by Visual Example. In: Pirotte, A., Delobel, C., Gottlob, G. (eds.) EDBT 1992. LNCS, vol. 580, pp. 56–71. Springer, Heidelberg (1992)CrossRefGoogle Scholar
  8. 8.
    Huiskes, M.J., Lew, M.S.: The MIR Flickr retrieval evaluation. In: ACM Int. Conf. on Multimedia Information Retrieval, MIR 2008 (2008)Google Scholar
  9. 9.
    Keysers, D., Deselaers, T., Gollan, C., Hermann, N.: Deformation models for image recognition. IEEE Trans. Pattern Analysis and Machine Intelligence 29(8), 1422–1435 (2007)CrossRefGoogle Scholar
  10. 10.
    Koutamanis, A.: Sketching with Digital Pen and Paper. In: Computer Aided Architectural Design Futures 2005 (June 2005)Google Scholar
  11. 11.
    Kurtenbach, G.: Pen-based computing. ACM Crossroads 16(4), 14–20 (2010)Google Scholar
  12. 12.
    Norrie, M.C., Signer, B., Weibel, N.: General framework for the rapid development of interactive paper applications. In: CoPADD 2006, Workshop on Collaborating Over Paper and Digital Documents, vol. 6, pp. 9–12 (2006)Google Scholar
  13. 13.
    Sikora, T.: The MPEG-7 Visual Standard for Content Description – an Overview. IEEE Trans. on Circuits & Systems for Video Technology 11(6), 696–702 (2001)MathSciNetCrossRefGoogle Scholar
  14. 14.
    Springmann, M., Al Kabary, I., Schuldt, H.: Experiences with QbS: Challenges and Evaluation of Known Image Search based on User-Drawn Sketches. CS Technical Report CS-2010-001, University of Basel, Switzerland (2010),
  15. 15.
    Springmann, M., Al Kabary, I., Schuldt, H.: Image retrieval at memory’s edge: Known image search based on user-drawn sketches. In: Conference on Information and Knowledge Management (CIKM 2010), pp. 1465–1468 (October 2010)Google Scholar
  16. 16.
    Veltkamp, R.C., Tanase, M.: Content-Based Image Retrieval Systems: A Survey. Technical Report UU-CS-2000-34, Utrecht University (2000)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Roman Kreuzer
    • 1
  • Michael Springmann
    • 1
  • Ihab Al Kabary
    • 1
  • Heiko Schuldt
    • 1
  1. 1.Databases and Information Systems Group, Department of Mathematics and Computer ScienceUniversity of BaselSwitzerland

Personalised recommendations