International Journal of Computer Vision

, Volume 108, Issue 1, pp 3–29

The Ignorant Led by the Blind: A Hybrid Human–Machine Vision System for Fine-Grained Categorization

  • Steve Branson
  • Grant Van Horn
  • Catherine Wah
  • Pietro Perona
  • Serge Belongie
Article

DOI: 10.1007/s11263-014-0698-4

Cite this article as:
Branson, S., Van Horn, G., Wah, C. et al. Int J Comput Vis (2014) 108: 3. doi:10.1007/s11263-014-0698-4

Abstract

We present a visual recognition system for fine-grained visual categorization. The system is composed of a human and a machine working together and combines the complementary strengths of computer vision algorithms and (non-expert) human users. The human users provide two heterogeneous forms of information object part clicks and answers to multiple choice questions. The machine intelligently selects the most informative question to pose to the user in order to identify the object class as quickly as possible. By leveraging computer vision and analyzing the user responses, the overall amount of human effort required, measured in seconds, is minimized. Our formalism shows how to incorporate many different types of computer vision algorithms into a human-in-the-loop framework, including standard multiclass methods, part-based methods, and localized multiclass and attribute methods. We explore our ideas by building a field guide for bird identification. The experimental results demonstrate the strength of combining ignorant humans with poor-sighted machines the hybrid system achieves quick and accurate bird identification on a dataset containing 200 bird species.

Keywords

Fine-grained categorization Human-in-the-loop Interactive Parts Attributes Crowdsourcing Deformable part models Pose mixture models Object recognition Information gain Birds 

Supplementary material

11263_2014_698_MOESM1_ESM.pdf (26 kb)
Supplementary material ESM1 (PDF 25.6 KB)
11263_2014_698_MOESM2_ESM.pdf (4.3 mb)
Supplementary material ESM2 (PDF 4.27 KB)

Supplementary material ESM3 (MP4 3.28 KB)

Copyright information

© Springer Science+Business Media New York 2014

Authors and Affiliations

  • Steve Branson
    • 1
  • Grant Van Horn
    • 2
  • Catherine Wah
    • 2
  • Pietro Perona
    • 1
  • Serge Belongie
    • 2
  1. 1.CaltechPasadenaUSA
  2. 2.University of California, San DiegoLa JollaUSA

Personalised recommendations