Machine Vision and Applications

, Volume 19, Issue 2, pp 105–123 | Cite as

Automated insect identification through concatenated histograms of local appearance features: feature vector generation and region detection for deformable objects

  • Natalia Larios
  • Hongli Deng
  • Wei Zhang
  • Matt Sarpola
  • Jenny Yuen
  • Robert Paasch
  • Andrew Moldenke
  • David A. Lytle
  • Salvador Ruiz Correa
  • Eric N. Mortensen
  • Linda G. Shapiro
  • Thomas G. Dietterich
Original Paper


This paper describes a computer vision approach to automated rapid-throughput taxonomic identification of stonefly larvae. The long-term objective of this research is to develop a cost-effective method for environmental monitoring based on automated identification of indicator species. Recognition of stonefly larvae is challenging because they are highly articulated, they exhibit a high degree of intraspecies variation in size and color, and some species are difficult to distinguish visually, despite prominent dorsal patterning. The stoneflies are imaged via an apparatus that manipulates the specimens into the field of view of a microscope so that images are obtained under highly repeatable conditions. The images are then classified through a process that involves (a) identification of regions of interest, (b) representation of those regions as SIFT vectors (Lowe, in Int J Comput Vis 60(2):91–110, 2004) (c) classification of the SIFT vectors into learned “features” to form a histogram of detected features, and (d) classification of the feature histogram via state-of-the-art ensemble classification algorithms. The steps (a) to (c) compose the concatenated feature histogram (CFH) method. We apply three region detectors for part (a) above, including a newly developed principal curvature-based region (PCBR) detector. This detector finds stable regions of high curvature via a watershed segmentation algorithm. We compute a separate dictionary of learned features for each region detector, and then concatenate the histograms prior to the final classification step. We evaluate this classification methodology on a task of discriminating among four stonefly taxa, two of which, Calineuria and Doroneuria, are difficult even for experts to discriminate. The results show that the combination of all three detectors gives four-class accuracy of 82% and three-class accuracy (pooling Calineuria and Doro-neuria) of 95%. Each region detector makes a valuable contribution. In particular, our new PCBR detector is able to discriminate Calineuria and Doroneuria much better than the other detectors.


Classification Object recognition Interest operators Region detectors SIFT descriptor 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Arbuckle, T., Schroder, S., Steinhage, V., Wittmann, D.: Biodiversity informatics in action: identification and monitoring of bee species using ABIS. In: Proceedings of the 15th International Symposium Informatics for Environmental Protection, vol. 1, pp. 425–430. Zurich (2001)Google Scholar
  2. 2.
    Bouchard, G., Triggs, B.: Hierarchical part-based visual object categorization. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. I 710–715 (2005). Scholar
  3. 3.
    Breiman, L.: Bagging predictors. Mach. Learn. 24(2), 123–140 (1996). Scholar
  4. 4.
    Breiman L., Friedman J., Olshen R. and Stone C. (1984). Classification and Regression Trees. Chapman and Hall, New York MATHGoogle Scholar
  5. 5.
    Burl, M., Weber, M., Perona, P.: A probabilistic approach to object recognition using local photometry and global geometry. In: Proceedings of the ECCV, pp. 628–641 (1998)Google Scholar
  6. 6.
    Carter J., Resh V., Hannaford M. and Myers M. (2006). Macroinvertebrates as biotic indicators of env. qual. In: Hauer, F. and Lamberti, G. (eds) Methods in Stream Ecology, pp 1–2. Academic, San Diego Google Scholar
  7. 7.
    Csurka, G., Dance, C., Fan, L., Williamowski, J., Bray, C.: Visual categorization with bags of keypoints. ECCV’04 workshop on Statistical Learning in Computer Vision, pp. 59–74 (2004)Google Scholar
  8. 8.
    Csurka, G., Bray, C., Fan, C.L.: Visual categorization with bags of keypoints. ECCV workshop (2004)Google Scholar
  9. 9.
    Do M., Harp J. and Norris K. (1999). A test of a pattern recognition system for identification of spiders. Bull. Entomol. Res. 89(3): 217–224 CrossRefGoogle Scholar
  10. 10.
    Dorko, G., Schmid, C.: Object class recognition using discriminative local features. INRIA—Rhone-Alpes, RR-5497, February, 2005, Rapport de recherche. 2005/DS05aGoogle Scholar
  11. 11.
    Dorkó, G., Schmid, C.: Object class recognition using discriminative local features (2005). Accepted under major revisions to IEEE Trans. Pattern Anal. Mach. Intell. (updated 13 September)Google Scholar
  12. 12.
    Fergus, R., Perona, P., Zisserman, A.: Object class recognition by unsupervised scale-invariant learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, vol.~2, pp. 264–271. Madison, Wisconsin (2003)Google Scholar
  13. 13.
    Freund, Y., Schapire, R.E.: Experiments with a new boosting algorithm. In: International Conference on Machine Learning, pp. 148–156 (1996). Scholar
  14. 14.
    Friedman, J., Hastie, T., Tibshirani, R.: Additive logistic regression: a statistical view of boosting (1998). edu/friedman98additive.htmlGoogle Scholar
  15. 15.
    Gaston K.J. and O’Neill M.A. (2004). Automated species identification: why not?. Philosophical Trans. R. Soc. B: Biol. Sci. 359(1444): 655–667 CrossRefGoogle Scholar
  16. 16.
    Harris, C., Stephens, M.: A combined corner and edge detector. Alvey Vision Conference, pp. 147–151 (1988)Google Scholar
  17. 17.
    Hilsenhoff W.L. (1988). Rapid field assessment of organic pollution with a family level biotic index. J. North Am. Benthol. Soc. 7: 65–68 CrossRefGoogle Scholar
  18. 18.
    Hopkins G.W. and Freckleton R.P. (2002). Declines in the numbers of amateur and professional taxonomists: implications for conservation. Anim. Conserv. 5(3): 245–249 CrossRefGoogle Scholar
  19. 19.
    Jurie F. and Schmid C. (2004). Scale-invariant shape features for recognition of object categories. CVPR 2: 90–96 Google Scholar
  20. 20.
    Jurie, F., Triggs, B.: Creating efficient codebooks for visual recognition. In: ICCV ’05: Proceedings of the 10th IEEE International Conference on Computer Vision (ICCV’05), vol. 1, pp. 604–610. IEEE Computer Society, Washington, DC, USA (2005). DOI Scholar
  21. 21.
    Kadir, T., Zisserman, A., Brady, M.: An affine invariant salient region detector. In: European Conference on Computer Vision (ECCV04), pp. 228–241 (2004)Google Scholar
  22. 22.
    Kumar, S., August, J., Hebert, M.: Exploiting inference for approximate parameter learning in discriminative fields: an empirical study. In: 5th International Workshop, EMMCVPR 2005, pp. 153–168. Springer, St. Augustine (2005)Google Scholar
  23. 23.
    Landwehr, N., Hall, M., Frank, E.: Logistic model trees. Mach. Learn. 59(1–2), 161–205 (2005). DOI 10.1007/s10994-005- 0466-3Google Scholar
  24. 24.
    Leibe, B., Seemann, E., Schiele, B.: Pedestrian detection in crowded scenes. In: CVPR ’05: Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), vol. 1, pp. 878–885. IEEE Computer Society, Washington, DC, USA (2005). DOI Scholar
  25. 25.
    Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60(2), 91–110 (2004). DOI 10.1023/B:VISI.0000029664.99615.94Google Scholar
  26. 26.
    Lucas, S.: Face recognition with continuous n-tuple classifier. In: Proceedings of the British Machine Vision Conference, pp. 222–231. Essex (1997)Google Scholar
  27. 27.
    Matas J., Chum O., Urban M. and Pajdla T. (2004). Robust wide- baseline stereo from maximally stable extremal regions. Image Vis. Comput. 22(10): 761–767 CrossRefGoogle Scholar
  28. 28.
    Mikolajczyk K. and Schmid C. (2002). An affine invariant interest point detector. ECCV 1(1): 128–142 Google Scholar
  29. 29.
    Mikolajczyk K. and Schmid C. (2004). Scale and affine invariant interest point detectors. IJCV 60(1): 63–86 CrossRefGoogle Scholar
  30. 30.
    Mikolajczyk, K., Tuytelaars, T., Schmid, C., Zisserman, A., Matas, J., Schaffalitzky, F., Kadir, T., Gool, L.V.: A comparison of affine region detectors. IJCV 65(1/2), 43–72 (2005). Scholar
  31. 31.
    O’Neill, M.A., Gauld, I.D., Gaston, K.J., Weeks, P.: Daisy: an automated invertebrate identification system using holistic vision techniques. In: Proceedings of the Inaugural Meeting BioNET-INTERNATIONAL Group for Computer-Aided Taxonomy (BIGCAT), pp. 13–22. Egham (2000)Google Scholar
  32. 32.
    Opelt, A., Fussenegger, M., Pinz, A., Auer, P.: Weak hypotheses and boosting for generic object detection and recognition. In: 8th European Conference on Computer Vision, vol. 2, pp. 71–84. Prague, Czech Republic (2004)Google Scholar
  33. 33.
    Opelt A., Pinz A., Fussenegger M. and Auer P. (2006). Generic object recognition with boosting. IEEE Trans. Pattern Anal. Mach. Intell. 28(3): 416–431 CrossRefGoogle Scholar
  34. 34.
    Papageorgiou C. and Poggio T. (2000). A trainable system for object detection. Int. J. Comput. Vis. 38(1): 15–33 MATHCrossRefGoogle Scholar
  35. 35.
    Quattoni, A., Collins, M., Darrell, T.: Conditional random fields for object recognition. In: Proceedings of the NIPS 2004. MIT Press, Cambridge (2005)Google Scholar
  36. 36.
    Quinlan J.R. (1993). C4.5: programs for machine learning. Morgan Kaufmann, San Francisco Google Scholar
  37. 37.
    Shotton, J., Blake, A., Cipolla, R.: Contour-based learning for object detection. In: ICCV ’05: Proceedings of the 10th IEEE International Conference on Computer Vision (ICCV’05), vol. 1, pp. 503–510. IEEE Computer Society, Washington, DC, USA (2005). DOI Scholar
  38. 38.
    Sokal, R.R., Rohlf, F.J.: Biometry, 3rd edn. W. H. Freeman, Gordonsville (1995)Google Scholar
  39. 39.
    Steger C. (1998). An unbiased detector of curvilinear structures. PAMI 20(2): 113–125 Google Scholar
  40. 40.
    Sung K.K. and Poggio T. (1998). Example-based learning for view-based human face detection. IEEE Trans. Pattern Anal. Mach. Intell. 20(1): 39–51 CrossRefGoogle Scholar
  41. 41.
    Turk, M.A., Pentland, A.P.: Face recognition using eigenfaces. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 586–591 (1991)Google Scholar
  42. 42.
    Tuytelaars, T., Gool, L.V.: Wide baseline stereo matching based on local, affinely invariant regions. BMVC, pp. 412–425 (2000)Google Scholar
  43. 43.
    Tuytelaars T. and Gool L.V. (2004). Matching widely separated views based on affine invariant regions. IJCV 59(1): 61–85 CrossRefGoogle Scholar
  44. 44.
    Vincent L. and Soille P. (1991). Watersheds in digital spaces: an efficient algorithm based on immersion simulations. PAMI 13(6): 583–598 Google Scholar
  45. 45.
    Zhang, W., Deng, H., Dietterich, T.G., Mortensen, E.N.: A hierarchical object recognition system based on multi-scale principal curvature regions. International Conference of Pattern Recognition, pp. 1475–1490 (2006)Google Scholar

Copyright information

© Springer-Verlag 2007

Authors and Affiliations

  • Natalia Larios
    • 1
  • Hongli Deng
    • 2
  • Wei Zhang
    • 2
  • Matt Sarpola
    • 3
  • Jenny Yuen
    • 6
  • Robert Paasch
    • 3
  • Andrew Moldenke
    • 4
  • David A. Lytle
    • 5
  • Salvador Ruiz Correa
    • 7
  • Eric N. Mortensen
    • 2
  • Linda G. Shapiro
    • 8
  • Thomas G. Dietterich
    • 2
  1. 1.Department of Electrical EngineeringUniversity of WashingtonSeattleUSA
  2. 2.School of Electrical Engineering and Computer ScienceOregon State UniversityCorvallisUSA
  3. 3.Department of Mechanical EngineeringOregon State UniversityCorvallisUSA
  4. 4.Department of Botany and Plant PathologyOregon State UniversityCorvallisUSA
  5. 5.Department of ZoologyOregon State UniversityCorvallisUSA
  6. 6.Computer Science and AI LaboratoryMassachusetts Institute of TechnologyCambridgeUSA
  7. 7.Department of Diagnostic Imaging and RadiologyChildren’s National Medical CenterWashingtonUSA
  8. 8.Department of Computer Science and EngineeringUniversity of WashingtonSeattleUSA

Personalised recommendations