Skip to main content
SpringerLink
Log in

Confidence Sets for Fine-Grained Categorization and Plant Species Identification

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

We present a new hierarchical strategy for fine-grained categorization. Standard, fully automated systems report a single estimate of the category, or perhaps a ranked list, but have non-neglible error rates for most realistic scenarios, which limits their utility. Instead, we propose a semi-automated system which outputs a it confidence set (CS)—a variable-length list of categories which contains the true one with high probability (e.g., a 99 % CS). Performance is then measured by the expected size of the CS, reflecting the effort required for final identification by the user. The implementation is based on a hierarchical clustering of the full set of categories. This tree of subsets provides a graded family of candidate CS’s containing visually similar categories. There is also a learned discriminant score for deciding between each subset and all others combined. Selection of the CS is based on the joint score likelihood under a Bayesian network model. We apply this method to determining the species of a plant from an image of a leaf against either a uniform or natural background. Extensive experiments are reported. We obtain superior results relative to existing methods for point estimates for scanned leaves and report the first useful results for natural images at the expense of asking the user to initialize the process by identifying specific landmarks.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24
Fig. 25
Fig. 26

Similar content being viewed by others

Notes

  1. http://www.imageclef.org/2011/Plants

  2. http://www.imageclef.org/2011/Plants

  3. http://www.tela-botanica.org

  4. http://www.imageclef.org/2011/Plants

  5. http://www.imageclef.org/2011/Plants

References

  • Angelova, A. & Zhu, S. (2013). Efficient object detection and segmentation for fine-grained recognition. In: CVPR.

  • Belhumeur, P.N., Chen, D., Feiner, S., Jacobs, D.W., Kress, W.J., Ling, H., Lopez, I.C., Ramamoorthi, R., Sheorey, S., White, S., Zhang, L. (2008). Searching the world’s herbaria: A system for visual identification of plant species. In: ECCV (4), pp 116–129.

  • Bourdev, L.D. & Malik, J. (2009). Poselets: Body part detectors trained using 3d human pose annotations. In: ICCV, pp 1365–1372.

  • Branson, S., Wah, C., Schroff, F., Babenko, B., Welinder, P., Perona, P. & Belongie, S. (2010). Visual recognition with humans in the loop. In: ECCV (4), pp 438–451.

  • Burl, M.C. & Perona, P. (1998). Using hierarchical shape models to spot keywords in cursive handwriting data. In: CVPR, pp 535–540.

  • Caballero, C. & Aranda, M.C. (2010). Plant species identification using leaf image retrieval. In: CIVR, pp 327–334.

  • Casanova, D., Florindo, J.B. & Bruno, O.M. (2011). Ifsc/usp at imageclef 2011: Plant identication task. In: CLEF (Notebook Papers/Labs/Workshop).

  • Casanova, D., Florindo, J.B., Gonçalves, W.N. & Bruno, O.M. (2012) Ifsc/usp at imageclef 2012: Plant identification task. In: CLEF (Online Working Notes/Labs/Workshop).

  • Cook, N. R. (2005). Confidence Intervals and Sets. : John Wiley and Sons Ltd.

  • Cope, J. S., Corney, D. P. A., Clark, J. Y., Remagnino, P., & Wilkin, P. (2012). Plant species identification using digital morphometrics: A review. Expert Syst Appl, 39(8), 7562–7573.

    Article  Google Scholar 

  • del Coz, J. J., Díez, J., & Bahamonde, A. (2009). Learning nondeterministic classifiers. Journal of Machine Learning Research, 10, 2273–2293.

    MATH  Google Scholar 

  • Deng, J., Berg, A.C., Li, K., Li, F.F. (2010). What does classifying more than 10, 000 image categories tell us? In: ECCV (5), pp 71–84.

  • Deng, J., Krause, J., Berg, A.C. & Li, F.F. (2012). Hedging your bets: Optimizing accuracy-specificity trade-offs in large scale visual recognition. In: CVPR, pp 3450–3457.

  • Deng, J., Krause, J. & Li, F.F. (2013). Fine-grained crowdsourcing for fine-grained recognition. In: CVPR, pp 580–587.

  • Du, J.X., Huang, D., Wang, X. & Gu, X. (2005). Shape recognition based on radial basis probabilistic neural network and application to plant species identification. In: ISNN (2), pp 281–285.

  • Duan, K., Parikh, D., Crandall, D.J. & Grauman, K. (2012). Discovering localized attributes for fine-grained recognition. In: CVPR, pp 3474–3481.

  • El-Yaniv, R., & Wiener, Y. (2010). On the foundations of noise-free selective classification. Journal of Machine Learning Research, 11, 1605–1641.

    MATH  MathSciNet  Google Scholar 

  • Ellis, B. (2009). Manual of leaf architecture. Cornell paperbacks, Published in association with the New York Botanical Garden.

  • Elpel, T. (2004). Botany in a Day: The Patterns Method of Plant Identification. Thomas J. Elpel’s herbal field guide to plant families of North America. : Hops Press.

  • Fan, X. & Geman, D. (2004). Hierarchical object indexing and sequential learning. In: ICPR (3), pp 65–68.

  • Farrell, R., Oza, O., Zhang, N., Morariu, V.I., Darrell, T., Davis, L.S. (2011a). Birdlets: Subordinate categorization using volumetric primitives and pose-normalized appearance. In: ICCV, pp 161–168.

  • Farrell, R., Oza, O., Zhang, Z., Morariu, V., Darrell, T. & Davis, L. (2011b). Birdlets: Subordinate categorization using volumetric primitives and pose-normalized appearance. In: ICCV, pp 161–168.

  • Felzenszwalb, P.F. & Schwartz, J.D. (2007). Hierarchical matching of deformable shapes. In: CVPR.

  • Ferecatu, M. (2005). Image retrieval with active relevance feedback using both visual and keyword-based descriptors. PhD thesis, Université de Versailles SaintQuentin-en-Yvelines.

  • Fergus, R., Bernal, H., Weiss, Y. & Torralba, A. (2010). Semantic label sharing for learning with many categories. In: ECCV (1), pp 762–775.

  • Fernández, A., & Gómez, S. (2008). Solving non-uniqueness in agglomerative hierarchical clustering using multidendrograms. J Classification, 25(1), 43–65.

    Article  MATH  MathSciNet  Google Scholar 

  • Goëau, H., Bonnet, P., Joly, A., Boujemaa, N., Barthelemy, D., Molino, J.F., Birnbaum, P., Mouysset, E. & Picard, M. (2011). The clef 2011 plant images classification task. In: CLEF (Notebook Papers/Labs/Workshop).

  • Goëau, H., Bonnet, P., Joly, A., Yahiaoui, I., Barthelemy, D., Boujemaa, N. & Molino, J.F. (2012). The imageclef 2012 plant identification task. In: CLEF (Online Working Notes/Labs/Workshop).

  • Grall-Maës, E., & Beauseroy, P. (2009). Optimal decision rule with class-selective rejection and performance constraints. IEEE Trans Pattern Anal Mach Intell, 31(11), 2073–2082.

    Article  Google Scholar 

  • Gu, X., Du, J.X. & Wang, X. (2005). Leaf recognition based on the combination of wavelet transform and gaussian interpolation. In: ICIC (1), pp 253–262.

  • Gupta, S. S. (1965). On some multiple decision (selection and ranking) rules. Technometrics, 7(2), 225–245.

    Article  MATH  Google Scholar 

  • Ha, T. M. (1997). The optimum class-selective rejection rule. IEEE Trans Pattern Anal Mach Intell, 19(6), 608–615.

    Article  Google Scholar 

  • Horiuchi, T. (1998). Class-selective rejection rule to minimize the maximum distance between selected classes. Pattern Recognition, 31(10), 1579–1588.

    Article  Google Scholar 

  • Jain, A. K., Murty, M. N., & Flynn, P. J. (1999). Data clustering: A review. ACM Comput Surv, 31(3), 264–323.

    Article  Google Scholar 

  • Jr, C. N. S., & Freitas, A. A. (2011). A survey of hierarchical classification across different application domains. Data Min Knowl Discov, 22(1–2), 31–72.

    MathSciNet  Google Scholar 

  • Kumar, N., Belhumeur, P.N., Biswas, A., Jacobs, D.W., Kress, W.J., Lopez, I.C. & Soares, J.V.B. (2012). Leafsnap: A computer vision system for automatic plant species identification. In: ECCV (2), pp 502–516.

  • Larios, N., Deng, H., Zhang, W., Sarpola, M., Yuen, J., Paasch, R., et al. (2008). Automated insect identification through concatenated histograms of local appearance features: feature vector generation and region detection for deformable objects. Mach Vis Appl, 19(2), 105–123.

    Article  Google Scholar 

  • Lazebnik, S., Schmid, C. & Ponce, J. (2006). Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories. In: CVPR (2), pp 2169–2178.

  • Lee, P. (1989). Bayesian statistics: an introduction. No. v. 2 in A Charles Griffin Book, Oxford University Press, http://books.google.fr/books?id=_hXvAAAAMAAJ

  • Li, F.F. & Perona, P. (2005). A bayesian hierarchical model for learning natural scene categories. In: CVPR (2), pp 524–531.

  • Ling, H., & Jacobs, D. W. (2007). Shape classification using the inner-distance. IEEE Trans Pattern Anal Mach Intell, 29(2), 286–299.

    Article  Google Scholar 

  • Liu, J., Kanazawa, A., Jacobs, D.W., Belhumeur, P.N. (2012). Dog breed classification using part localization. In: ECCV (1), pp 172–185.

  • Manh, A. G., Rabatel, G., Assemat, L., & Aldon, M. J. (2001). Weed leaf image segmentation by deformable templates. Journal of agricultural engineering research, 80(2), 139–146.

    Article  Google Scholar 

  • Martínez-Muñoz, G., Delgado, N.L., Mortensen, E.N., Zhang, W., Yamamuro, A., Paasch, R., Payet, N., Lytle, D.A., Shapiro, L.G., Todorovic, S., Moldenke, A. & Dietterich, T.G. (2009). Dictionary-free categorization of very similar objects via stacked evidence trees. In: CVPR, pp 549–556.

  • Mouine, S., Yahiaoui, I., Verroust-Blondet, A. (2013). A shape-based approach for leaf classification using multiscaletriangular representation. In: ICMR, pp 127–134.

  • Neyman, J. (1937). Outline of a theory of statistical estimation based on the classical theory of probability. Philosophical Transactions of the Royal Society of London Series A, Mathematical and Physical Sciences 236(767):pp. 333–380, http://www.jstor.org/stable/91337

  • Nilsback, M.E. & Zisserman, A. (2006). A visual vocabulary for flower classification. In: CVPR (2), pp 1447–1454.

  • Otsu, N. (1979). A Threshold Selection Method from Gray-level Histograms. Man and Cybernetics: IEEE Transactions on Systems.

    Google Scholar 

  • Rejeb Sfar, A., Boujemaa, N. & Geman, D. (2013a). Identification of plants from multiple images and botanical idkeys. In: ICMR, pp 191–198.

  • Rejeb Sfar, A., Boujemaa, N., Geman, D. (2013b). Vantage feature frames for fine-grained categorization. In: CVPR, pp 835–842.

  • Söderkvist, O. (2001). Computer vision classification of leaves from swedish trees. Master’s thesis, Linköping University, SE-581 83 Linköping, Sweden, liTH-ISY-EX-3132.

  • Teng, C.H., Kuo, Y.T. & Chen, Y.S. (2009). Leaf segmentation, its 3d position estimation and leaf classification from a few images with very close viewpoints. In: ICIAR, pp 937–946.

  • Tversky, B. & Hemenway, K. (1984). Objects, parts, and categories. Experimental Psychology: General.

  • Wah, C., Branson, S., Perona, P. & Belongie, S. (2011). Multiclass recognition and part localization with humans in the loop. In: ICCV, pp 2524–2531.

  • Wang, J., Yang, J., Yu, K., Lv, F., Huang, T. & Gong, Y. (2010). Locality-constrained linear coding for image classification. In: CVPR, pp 3360–3367.

  • Wang, X., Du, J.X. & Zhang, G.J. (2005). Recognition of leaf images based on shape features using a hypersphere classifier. In: ICIC (1), pp 87–96.

  • Wang, X. F., Huang, D. S., Du, J. X., Xu, H., & Heutte, L. (2008). Classification of plant leaf images with complicated background. Applied Mathematics and Computation, 205(2), 916–926.

    Article  MATH  MathSciNet  Google Scholar 

  • Ward, J, Jr. (1963). Hierarchical grouping to optimize an objective function. Journal of the American Statistical Association, 58(301), 236–244.

    Article  MathSciNet  Google Scholar 

  • Wu, J. & Rehg, J.M. (2008). Where am i: Place instance and category recognition using spatial pact. In: CVPR.

  • Wu, S.G., Bao, F.S., Xu, E.Y., Wang, Y., Chang, Y.F. & Xiang, Q.L. (2007). A leaf recognition algorithm for plant classification using probabilistic neural network. CoRR abs/0707.4289.

  • Yang, S., Bo, L., Wang, J. & Shapiro, L.G. (2012). Unsupervised template learning for fine-grained object recognition. In: NIPS, pp 3131–3139.

  • Yao, B., Bradski, G.R. & Li, F.F. (2012). A codebook-free and annotation-free approach for fine-grained image categorization. In: CVPR, pp 3466–3473.

  • Yuan, M., & Wegkamp, M. H. (2010). Classification methods with reject option based on convex risk minimization. Journal of Machine Learning Research, 11, 111–130.

    MATH  MathSciNet  Google Scholar 

  • Zhang, N., Farrell, R. & Darrell, T. (2012). Pose pooling kernels for sub-category recognition. In: CVPR, pp 3665–3672.

  • Zweig, A. & Weinshall, D. (2007). Exploiting object hierarchy: Combining models from different category levels. In: ICCV, pp 1–8.

Download references

Author information

Authors and Affiliations

  1. INRIA Saclay, Palaiseau, France

    Asma Rejeb Sfar & Nozha Boujemaa

  2. Johns Hopkins University, Baltimore, MD, USA

    Donald Geman

Authors
  1. Asma Rejeb Sfar
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nozha Boujemaa
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Donald Geman
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Asma Rejeb Sfar.

Additional information

Communicated by Derek Hoiem.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 611 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Rejeb Sfar, A., Boujemaa, N. & Geman, D. Confidence Sets for Fine-Grained Categorization and Plant Species Identification. Int J Comput Vis 111, 255–275 (2015). https://doi.org/10.1007/s11263-014-0743-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-014-0743-3

Keywords

Search

Navigation