Advertisement

Journal on Multimodal User Interfaces

, Volume 2, Issue 1, pp 3–11 | Cite as

MARQS: retrieving sketches learned from a single example using a dual-classifier

  • Brandon PaulsonEmail author
  • Tracy Hammond
Original Paper

Abstract

Mouse and keyboard interfaces handle traditional text-based queries, and standard search engines provide for effective text-based search. However, everyday documents are filled with not only text, but photos, cartoons, diagrams, and sketches. These images can often be easier to recall than the surrounding text. In an effort to make human computer interaction handle more forms of human-human interaction, sketching has recently become an important means of interacting with computer systems. We propose extending the traditional monomodal model of text-based search to include the capabilities of sketch-based search. Our goal is to create a sketch-based search that can find documents from a single query sketch. We imagine an important use for this technology would be to allow users to search a computerized laboratory notebook for a previously drawn sketch. Because such as sketch will have initially been drawn only a single time, it is important that the search-by-sketch system (1) recognize a wide range of shapes that are not necessarily geometric nor drawn in the same way each time, (2) recognize a query example from only one initial training example, and (3) learn from successful queries to improve accuracy over time. We present here such an algorithm. To test the algorithm, we implemented a proof-of-concept-system: MARQS, a system that uses sketches to query existing media albums. Preliminary results show that the system yielded an average search rank of 1.51, indicating that the correct sketch is presented as either the top or second search result on average.

Keywords

Sketch recognition Sketch-based interfaces Search by sketch Feature-based recognition 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Gross MD, Do EY-L (1996) Ambiguous intentions: a paper-like interface for creative design. In: UIST ’96: proceedings of the 9th annual ACM symposium on user interface software and technology. ACM, New York, pp 183–192 CrossRefGoogle Scholar
  2. 2.
    Gross MD, Do EYL (1996) Demonstrating the electronic cocktail napkin: a paper-like interface for early design. In: CHI ’96: conference companion on human factors in computing systems. ACM, New York, pp 5–6 CrossRefGoogle Scholar
  3. 3.
    Hammond T, Davis R (2005) Ladder, a sketching language for user interface developers. Comput Graph 29(4):518–532 CrossRefGoogle Scholar
  4. 4.
    Long AC, Landay JA, Rowe LA (2001) “Those look similar!” issues in automating gesture design advice. In: PUI ’01: proceedings of the 2001 workshop on perceptive user interfaces. ACM, New York, pp 1–5 CrossRefGoogle Scholar
  5. 5.
    Morrel-Samuels P (1990) Clarifying the distinction between lexical and gestural commands. Int J Man-Mach Stud 32(5):581–590 CrossRefGoogle Scholar
  6. 6.
    Chris Long JA, Landay JA, Rowe LA, Michiels J (2000) Visual similarity of pen gestures. In: CHI ’00: proceedings of the SIGCHI conference on human factors in computing systems. ACM, New York, pp 360–367 CrossRefGoogle Scholar
  7. 7.
    Rubine D (1991) Specifying gestures by example. In: SIGGRAPH ’91: proceedings of the 18th annual conference on computer graphics and interactive techniques. ACM, New York, pp 329–337 CrossRefGoogle Scholar
  8. 8.
    Fonseca MJ, Pimentel C, Jorge JA (2002) Cali: an online scribble recognizer for calligraphic interfaces, In: AAAI spring symposium on sketch understanding, pp 51–58 Google Scholar
  9. 9.
    Kara LB, Stahovich TF (2004) An image-based trainable symbol recognizer for sketch-based interfaces. In: AAAI fall symposium series 2004: making pen-based interaction intelligent and natural, pp 99–105 Google Scholar
  10. 10.
    Sun Z, Jiang E, Sun J (2005) Adaptive online multi-stroke sketch recognition based on hidden Markov model. In: Lecture notes in artificial intelligences, vol 3784. Springer, Berlin, pp 948–957 Google Scholar
  11. 11.
    Sutherland IE (1988) Sketchpad a man-machine graphical communication system. In: 25 years of DAC: papers on twenty-five years of electronic design automation. ACM, New York, pp 507–524 CrossRefGoogle Scholar
  12. 12.
    Sezgin TM, Stahovich T, Davis R (2001) Sketch based interfaces: early processing for sketch understanding. In: PUI ’01: proceedings of the 2001 workshop on perceptive user interfaces. ACM, New York, pp 1–8 CrossRefGoogle Scholar
  13. 13.
    Yu B, Cai S (2003) A domain-independent system for sketch recognition. In: GRAPHITE ’03: proceedings of the 1st international conference on computer graphics and interactive techniques in Australasia and South East Asia. ACM, New York, pp 141–146 CrossRefGoogle Scholar
  14. 14.
    Paulson B, Hammond T (2008) Paleosketch: Accurate primitive sketch recognition and beautification. In: IUI ’08: proceedings of the 2008 international conference on intelligent user interfaces. pp 1–8 Google Scholar
  15. 15.
    Kato T, Kurita T, Otsu N, Hirata K (1992) A sketch retrieval method for full color image databases—query by visual example. In: 11th IAPA international conference on pattern recognition, pp 530–533 Google Scholar
  16. 16.
    Gennari LM, Kara LB, Stahovich TF (2004) Combining geometry and domain knowledge to interpret hand-drawn diagrams. In: AAAI fall symposium series 2004: making pen-based interaction intelligent and natural, pp 547–562 Google Scholar
  17. 17.
    Leung WH, Chen T (2002) Retrieval of sketches based on spatial relation between strokes. In: Proceedings of the 2002 international conference on image processing, pp I-908–I-911 Google Scholar
  18. 18.
    Saund E, Fleet D, Larner D, Mahoney J (2003) Perceptually-supported image editing of text and graphics. In: UIST ’03: proceedings of the 16th annual ACM symposium on user interface software and technology. ACM, New York, pp 183–192 CrossRefGoogle Scholar

Copyright information

© OpenInterface Association 2008

Authors and Affiliations

  1. 1.Sketch Recognition LabTexas A&M UniversityCollege StationUSA

Personalised recommendations