International Journal of Computer Vision

, Volume 111, Issue 3, pp 255–275 | Cite as

Confidence Sets for Fine-Grained Categorization and Plant Species Identification

  • Asma Rejeb SfarEmail author
  • Nozha Boujemaa
  • Donald Geman


We present a new hierarchical strategy for fine-grained categorization. Standard, fully automated systems report a single estimate of the category, or perhaps a ranked list, but have non-neglible error rates for most realistic scenarios, which limits their utility. Instead, we propose a semi-automated system which outputs a it confidence set (CS)—a variable-length list of categories which contains the true one with high probability (e.g., a 99 % CS). Performance is then measured by the expected size of the CS, reflecting the effort required for final identification by the user. The implementation is based on a hierarchical clustering of the full set of categories. This tree of subsets provides a graded family of candidate CS’s containing visually similar categories. There is also a learned discriminant score for deciding between each subset and all others combined. Selection of the CS is based on the joint score likelihood under a Bayesian network model. We apply this method to determining the species of a plant from an image of a leaf against either a uniform or natural background. Extensive experiments are reported. We obtain superior results relative to existing methods for point estimates for scanned leaves and report the first useful results for natural images at the expense of asking the user to initialize the process by identifying specific landmarks.


Fine-grained categorization Hierarchical representation Confidence set Plant identification Semi-automated 

Supplementary material

11263_2014_743_MOESM1_ESM.pdf (612 kb)
Supplementary material 1 (pdf 611 KB)


  1. Angelova, A. & Zhu, S. (2013). Efficient object detection and segmentation for fine-grained recognition. In: CVPR.Google Scholar
  2. Belhumeur, P.N., Chen, D., Feiner, S., Jacobs, D.W., Kress, W.J., Ling, H., Lopez, I.C., Ramamoorthi, R., Sheorey, S., White, S., Zhang, L. (2008). Searching the world’s herbaria: A system for visual identification of plant species. In: ECCV (4), pp 116–129.Google Scholar
  3. Bourdev, L.D. & Malik, J. (2009). Poselets: Body part detectors trained using 3d human pose annotations. In: ICCV, pp 1365–1372.Google Scholar
  4. Branson, S., Wah, C., Schroff, F., Babenko, B., Welinder, P., Perona, P. & Belongie, S. (2010). Visual recognition with humans in the loop. In: ECCV (4), pp 438–451.Google Scholar
  5. Burl, M.C. & Perona, P. (1998). Using hierarchical shape models to spot keywords in cursive handwriting data. In: CVPR, pp 535–540.Google Scholar
  6. Caballero, C. & Aranda, M.C. (2010). Plant species identification using leaf image retrieval. In: CIVR, pp 327–334.Google Scholar
  7. Casanova, D., Florindo, J.B. & Bruno, O.M. (2011). Ifsc/usp at imageclef 2011: Plant identication task. In: CLEF (Notebook Papers/Labs/Workshop).Google Scholar
  8. Casanova, D., Florindo, J.B., Gonçalves, W.N. & Bruno, O.M. (2012) Ifsc/usp at imageclef 2012: Plant identification task. In: CLEF (Online Working Notes/Labs/Workshop).Google Scholar
  9. Cook, N. R. (2005). Confidence Intervals and Sets. : John Wiley and Sons Ltd.Google Scholar
  10. Cope, J. S., Corney, D. P. A., Clark, J. Y., Remagnino, P., & Wilkin, P. (2012). Plant species identification using digital morphometrics: A review. Expert Syst Appl, 39(8), 7562–7573.CrossRefGoogle Scholar
  11. del Coz, J. J., Díez, J., & Bahamonde, A. (2009). Learning nondeterministic classifiers. Journal of Machine Learning Research, 10, 2273–2293.zbMATHGoogle Scholar
  12. Deng, J., Berg, A.C., Li, K., Li, F.F. (2010). What does classifying more than 10, 000 image categories tell us? In: ECCV (5), pp 71–84.Google Scholar
  13. Deng, J., Krause, J., Berg, A.C. & Li, F.F. (2012). Hedging your bets: Optimizing accuracy-specificity trade-offs in large scale visual recognition. In: CVPR, pp 3450–3457.Google Scholar
  14. Deng, J., Krause, J. & Li, F.F. (2013). Fine-grained crowdsourcing for fine-grained recognition. In: CVPR, pp 580–587.Google Scholar
  15. Du, J.X., Huang, D., Wang, X. & Gu, X. (2005). Shape recognition based on radial basis probabilistic neural network and application to plant species identification. In: ISNN (2), pp 281–285.Google Scholar
  16. Duan, K., Parikh, D., Crandall, D.J. & Grauman, K. (2012). Discovering localized attributes for fine-grained recognition. In: CVPR, pp 3474–3481.Google Scholar
  17. El-Yaniv, R., & Wiener, Y. (2010). On the foundations of noise-free selective classification. Journal of Machine Learning Research, 11, 1605–1641.zbMATHMathSciNetGoogle Scholar
  18. Ellis, B. (2009). Manual of leaf architecture. Cornell paperbacks, Published in association with the New York Botanical Garden.Google Scholar
  19. Elpel, T. (2004). Botany in a Day: The Patterns Method of Plant Identification. Thomas J. Elpel’s herbal field guide to plant families of North America. : Hops Press.Google Scholar
  20. Fan, X. & Geman, D. (2004). Hierarchical object indexing and sequential learning. In: ICPR (3), pp 65–68.Google Scholar
  21. Farrell, R., Oza, O., Zhang, N., Morariu, V.I., Darrell, T., Davis, L.S. (2011a). Birdlets: Subordinate categorization using volumetric primitives and pose-normalized appearance. In: ICCV, pp 161–168.Google Scholar
  22. Farrell, R., Oza, O., Zhang, Z., Morariu, V., Darrell, T. & Davis, L. (2011b). Birdlets: Subordinate categorization using volumetric primitives and pose-normalized appearance. In: ICCV, pp 161–168.Google Scholar
  23. Felzenszwalb, P.F. & Schwartz, J.D. (2007). Hierarchical matching of deformable shapes. In: CVPR.Google Scholar
  24. Ferecatu, M. (2005). Image retrieval with active relevance feedback using both visual and keyword-based descriptors. PhD thesis, Université de Versailles SaintQuentin-en-Yvelines.Google Scholar
  25. Fergus, R., Bernal, H., Weiss, Y. & Torralba, A. (2010). Semantic label sharing for learning with many categories. In: ECCV (1), pp 762–775.Google Scholar
  26. Fernández, A., & Gómez, S. (2008). Solving non-uniqueness in agglomerative hierarchical clustering using multidendrograms. J Classification, 25(1), 43–65.CrossRefzbMATHMathSciNetGoogle Scholar
  27. Goëau, H., Bonnet, P., Joly, A., Boujemaa, N., Barthelemy, D., Molino, J.F., Birnbaum, P., Mouysset, E. & Picard, M. (2011). The clef 2011 plant images classification task. In: CLEF (Notebook Papers/Labs/Workshop).Google Scholar
  28. Goëau, H., Bonnet, P., Joly, A., Yahiaoui, I., Barthelemy, D., Boujemaa, N. & Molino, J.F. (2012). The imageclef 2012 plant identification task. In: CLEF (Online Working Notes/Labs/Workshop).Google Scholar
  29. Grall-Maës, E., & Beauseroy, P. (2009). Optimal decision rule with class-selective rejection and performance constraints. IEEE Trans Pattern Anal Mach Intell, 31(11), 2073–2082.CrossRefGoogle Scholar
  30. Gu, X., Du, J.X. & Wang, X. (2005). Leaf recognition based on the combination of wavelet transform and gaussian interpolation. In: ICIC (1), pp 253–262.Google Scholar
  31. Gupta, S. S. (1965). On some multiple decision (selection and ranking) rules. Technometrics, 7(2), 225–245.CrossRefzbMATHGoogle Scholar
  32. Ha, T. M. (1997). The optimum class-selective rejection rule. IEEE Trans Pattern Anal Mach Intell, 19(6), 608–615.CrossRefGoogle Scholar
  33. Horiuchi, T. (1998). Class-selective rejection rule to minimize the maximum distance between selected classes. Pattern Recognition, 31(10), 1579–1588.CrossRefGoogle Scholar
  34. Jain, A. K., Murty, M. N., & Flynn, P. J. (1999). Data clustering: A review. ACM Comput Surv, 31(3), 264–323.CrossRefGoogle Scholar
  35. Jr, C. N. S., & Freitas, A. A. (2011). A survey of hierarchical classification across different application domains. Data Min Knowl Discov, 22(1–2), 31–72.MathSciNetGoogle Scholar
  36. Kumar, N., Belhumeur, P.N., Biswas, A., Jacobs, D.W., Kress, W.J., Lopez, I.C. & Soares, J.V.B. (2012). Leafsnap: A computer vision system for automatic plant species identification. In: ECCV (2), pp 502–516.Google Scholar
  37. Larios, N., Deng, H., Zhang, W., Sarpola, M., Yuen, J., Paasch, R., et al. (2008). Automated insect identification through concatenated histograms of local appearance features: feature vector generation and region detection for deformable objects. Mach Vis Appl, 19(2), 105–123.CrossRefGoogle Scholar
  38. Lazebnik, S., Schmid, C. & Ponce, J. (2006). Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories. In: CVPR (2), pp 2169–2178.Google Scholar
  39. Lee, P. (1989). Bayesian statistics: an introduction. No. v. 2 in A Charles Griffin Book, Oxford University Press,
  40. Li, F.F. & Perona, P. (2005). A bayesian hierarchical model for learning natural scene categories. In: CVPR (2), pp 524–531.Google Scholar
  41. Ling, H., & Jacobs, D. W. (2007). Shape classification using the inner-distance. IEEE Trans Pattern Anal Mach Intell, 29(2), 286–299.CrossRefGoogle Scholar
  42. Liu, J., Kanazawa, A., Jacobs, D.W., Belhumeur, P.N. (2012). Dog breed classification using part localization. In: ECCV (1), pp 172–185.Google Scholar
  43. Manh, A. G., Rabatel, G., Assemat, L., & Aldon, M. J. (2001). Weed leaf image segmentation by deformable templates. Journal of agricultural engineering research, 80(2), 139–146.CrossRefGoogle Scholar
  44. Martínez-Muñoz, G., Delgado, N.L., Mortensen, E.N., Zhang, W., Yamamuro, A., Paasch, R., Payet, N., Lytle, D.A., Shapiro, L.G., Todorovic, S., Moldenke, A. & Dietterich, T.G. (2009). Dictionary-free categorization of very similar objects via stacked evidence trees. In: CVPR, pp 549–556.Google Scholar
  45. Mouine, S., Yahiaoui, I., Verroust-Blondet, A. (2013). A shape-based approach for leaf classification using multiscaletriangular representation. In: ICMR, pp 127–134.Google Scholar
  46. Neyman, J. (1937). Outline of a theory of statistical estimation based on the classical theory of probability. Philosophical Transactions of the Royal Society of London Series A, Mathematical and Physical Sciences 236(767):pp. 333–380,
  47. Nilsback, M.E. & Zisserman, A. (2006). A visual vocabulary for flower classification. In: CVPR (2), pp 1447–1454.Google Scholar
  48. Otsu, N. (1979). A Threshold Selection Method from Gray-level Histograms. Man and Cybernetics: IEEE Transactions on Systems.Google Scholar
  49. Rejeb Sfar, A., Boujemaa, N. & Geman, D. (2013a). Identification of plants from multiple images and botanical idkeys. In: ICMR, pp 191–198.Google Scholar
  50. Rejeb Sfar, A., Boujemaa, N., Geman, D. (2013b). Vantage feature frames for fine-grained categorization. In: CVPR, pp 835–842.Google Scholar
  51. Söderkvist, O. (2001). Computer vision classification of leaves from swedish trees. Master’s thesis, Linköping University, SE-581 83 Linköping, Sweden, liTH-ISY-EX-3132. Google Scholar
  52. Teng, C.H., Kuo, Y.T. & Chen, Y.S. (2009). Leaf segmentation, its 3d position estimation and leaf classification from a few images with very close viewpoints. In: ICIAR, pp 937–946.Google Scholar
  53. Tversky, B. & Hemenway, K. (1984). Objects, parts, and categories. Experimental Psychology: General.Google Scholar
  54. Wah, C., Branson, S., Perona, P. & Belongie, S. (2011). Multiclass recognition and part localization with humans in the loop. In: ICCV, pp 2524–2531.Google Scholar
  55. Wang, J., Yang, J., Yu, K., Lv, F., Huang, T. & Gong, Y. (2010). Locality-constrained linear coding for image classification. In: CVPR, pp 3360–3367.Google Scholar
  56. Wang, X., Du, J.X. & Zhang, G.J. (2005). Recognition of leaf images based on shape features using a hypersphere classifier. In: ICIC (1), pp 87–96.Google Scholar
  57. Wang, X. F., Huang, D. S., Du, J. X., Xu, H., & Heutte, L. (2008). Classification of plant leaf images with complicated background. Applied Mathematics and Computation, 205(2), 916–926.CrossRefzbMATHMathSciNetGoogle Scholar
  58. Ward, J, Jr. (1963). Hierarchical grouping to optimize an objective function. Journal of the American Statistical Association, 58(301), 236–244.CrossRefMathSciNetGoogle Scholar
  59. Wu, J. & Rehg, J.M. (2008). Where am i: Place instance and category recognition using spatial pact. In: CVPR.Google Scholar
  60. Wu, S.G., Bao, F.S., Xu, E.Y., Wang, Y., Chang, Y.F. & Xiang, Q.L. (2007). A leaf recognition algorithm for plant classification using probabilistic neural network. CoRR abs/0707.4289.Google Scholar
  61. Yang, S., Bo, L., Wang, J. & Shapiro, L.G. (2012). Unsupervised template learning for fine-grained object recognition. In: NIPS, pp 3131–3139.Google Scholar
  62. Yao, B., Bradski, G.R. & Li, F.F. (2012). A codebook-free and annotation-free approach for fine-grained image categorization. In: CVPR, pp 3466–3473.Google Scholar
  63. Yuan, M., & Wegkamp, M. H. (2010). Classification methods with reject option based on convex risk minimization. Journal of Machine Learning Research, 11, 111–130.zbMATHMathSciNetGoogle Scholar
  64. Zhang, N., Farrell, R. & Darrell, T. (2012). Pose pooling kernels for sub-category recognition. In: CVPR, pp 3665–3672.Google Scholar
  65. Zweig, A. & Weinshall, D. (2007). Exploiting object hierarchy: Combining models from different category levels. In: ICCV, pp 1–8.Google Scholar

Copyright information

© Springer Science+Business Media New York 2014

Authors and Affiliations

  1. 1.INRIA SaclayPalaiseauFrance
  2. 2.Johns Hopkins UniversityBaltimoreUSA

Personalised recommendations