Skip to main content

Advertisement

SpringerLink
Log in

Habitat image annotation with low-level features, medium-level knowledge and location information

  • Special Issue Paper
  • Published:
Multimedia Systems Aims and scope Submit manuscript

Abstract

The classification of habitats is crucial for structuring knowledge and developing our understanding of the natural world. Currently, most successful methods employ human surveyors—a laborious, expensive and subjective process. In this paper, we formulate habitat classification as a fine-grained visual categorization problem. We build on previous work and propose an image annotation framework that uses a novel automatic random forest-based method and that takes into consideration visual and geographical closeness in the classification process. During training, low-level visual features and medium-level contextual information are extracted. For the latter, we use a human-in-the-loop methodology by asking humans a set of 17 questions about the appearances of the image that can be easily answered by non-ecologists to extract medium-level knowledge about the images. During testing, and considering that close areas have similar ecological properties, we weigh the influence of the prediction of each tree of the forest according to their distance to the unseen test photography. Additionally, we present an updated version of a geo-referenced habitat image database containing over 1,000 high-resolution ground photographs that have been manually annotated by habitat classification experts. This has been made publicly available image database specifically designed for the development of multimedia analysis techniques for ecological applications. We show experimental recall and precision results which illustrate that our image annotation framework is able to annotate with a reasonable degree of confidence four of the main habitat classes: woodland and scrub, grassland and marsh, heathland and miscellaneous.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Notes

  1. http://www.viplab.cs.nott.ac.uk/download/habitat_classification_database.html.

References

  1. Andreson, J.R., Hardy, E.E., Roach, J.T., Witmer, R.E.: A Land Use and Land Cover Classification System for Use with Remote Sensing Data, p. 964. US Geological Survey professional paper (1976)

  2. Bargiel, D.: Capabilities of high resolution satellite radar for the detection of semi-natural habitat structures and grasslands in agricultural landscapes. Ecol.Inform. 13, 9–16 (2013)

    Article  Google Scholar 

  3. Branson, S., Horn, G., Wah, C., Perona, P., Belongie, S.: The ignorant led by the blind: a hybrid human–machine vision system for fine-grained categorization. Int. J. Comput. Vis. (IJCV), 1–27 (2014)

  4. Branson, S., Perona, P., Belongie, S.: Strong Supervision from Weak Annotation: Interactive Training of Deformable Part Models, IEEE International Conference on Computer Vision (ICCV 2011), 1832–1839 (2011)

  5. Branson, S., Wah, C., Schroff, F., Babenko, B., Welinder, P., Perona, P., Belongie, S.: Visual Recognition With Humans in the Loop, The 11th European Conference on Computer Vision (ECCV 2010), 438–451 (2010)

  6. Bingham, E, Mannila, H.: Random Projection in Dimensionality Reduction: Applications to Image and Text Data, In: Proceedings of the Seventh ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD ‘01). ACM, New York, NY, USA, pp. 245–250. http://doi.acm.org/10.1145/502512.502546s (2001). Accessed Apr 2014

  7. Bock, M.: Remote sensing and GIS-based techniques for the classification and monitoring of biotopes: case examples for a wet grass- and moor land area in Northern Germany. J. Nat. Conserv. 3, 145–155 (2003)

    Google Scholar 

  8. Boyd, D.S., Sanchez-Hernandez, C., Foody, G.M.: Mapping a specific class for priority habitats monitoring from satellite sensor data. Int. J. Remote Sens. 27, 2631–2644 (2006)

    Article  Google Scholar 

  9. Breiman, L.: Random Forests. Mach. Learn. 45(1), 5–32 (2001)

    Article  MathSciNet  MATH  Google Scholar 

  10. Bronaugh, W.F.J.: ‘Human-in-the-loop’ simulation: the right tool for port design. Port Technol. Int. 32, 1–2 (2007)

    Google Scholar 

  11. Chen, L.C., Rau, J.Y.: Detection of shoreline changes for tideland areas using multi-temporal satellite images. Int. J. Remote Sens. 19(17), 3383–3397 (1997)

    Article  Google Scholar 

  12. Council Directive 92/43/EEC of 21 May of 1992 on the Conservation of Natural Habitats and of Wild Flora and Fauna, OJ L 206, pp. 7–50 (1992)

  13. Cowardin, L.M., Carter, V., Golet, F.C., LaRoe, E.T.: Classification of Wetlands and Deepwater Habitats of the United States. US Department of the Interior, Fish and Wildlife Service, Washington, DC. http://www.npwrc.usgs.gov/resource/wetlands/classwet/index.htm (1979). Accessed Apr 2014

  14. Criminisi, A., Shotton, J., Konukoglu, E.: Decision Forests for Classification, Regression, Density Estimation, Manifold Learning and Semi-Supervised Learning. Tech. Rep. MSR-TR-2011-114, Microsoft Research, Cambridge. http://research.microsoft.com/pubs/155552/decisionForests_MSR_TR_2011_114.pdf (2011). Accessed Apr 2014

  15. Diaz-Varela, R.A., Ramil Rego, P., Iglesias, S.C., Muñoz Sobrino, C.: Automatic habitat classification methods based on satellite images: a practical assessment in the NW Iberia coastal mountains. Environ. Monit. Assess. 144(1–3), 229–250 (2008)

    Article  Google Scholar 

  16. Fu, H., Q., Zhang, Qiu, G.: Random Forest for Image Annotation. The 12th European Conference on Computer Vision (ECCV 2012), pp. 86–99 (2012)

  17. Gislason, P.O., Benediktsson, J.A., Sveinsson, J.R.: Random forests for land cover classification. Pattern Recogn. Lett. 27(4), 294–300 (2006)

    Article  Google Scholar 

  18. JNCC.: Handbook for Phase 1 Habitat Survey—A Technique for Environmental Audit. http://jncc.defra.gov.uk/PDF/pub10_handbookforphase1habitatsurvey.pdf (2010). Accessed Apr 2014

  19. Kontschieder, P., Bulo, S.R., Bischof, H., Pelillo, M.: Structured Class-Labels in Random Forests for Semantic Image Labelling, IEEE International Conference on Computer Vision (ICCV), 2011, pp. 2190–2197, 6–13, Nov 2011

  20. Korc, F., Schneider S. 2007. Annotation Tool. Technical report TR-IGG-P-2007-01, University of Bonn, Department of Photogrammetry

  21. Kumar, N., Belhumeur, P., Biswas, A., Jacobs,D., Kress,W., Lopez, I., Soares, J.: Leafsnap: A Computer Vision System for Automatic Plant Species Identification. In: ECCV, pp. 502-516. Springer, Heidelberg (2012)

  22. Lazebnik, S., Schmid, C., Ponce, J.: Beyond bags of features: spatial pyramid matching for recognizing natural scene categories, IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. 2(1), 2169–2178 (2006)

  23. Lauver, C.L., Whistler, J.L.: A hierarchical classification of Landsat TM imagery to identify natural grassland areas and rare species habitat. Photogramm. Eng. Remote Sens. 59(5), 627–634 (1993)

    Google Scholar 

  24. Liu, J., Kanazawa, A., Jacobs, D., Belhumeur, P.: Dog Breed Classification Using Part Localization. In Computer Vision–ECCV 2012, pp. 172–185. Springer, Heidelberg (2012)

  25. Lucas, R., Medcalf, K., Brown, A., Bunting, P., Breyer, J., Clewley, D., Keyworth, S., Blackmore, P.: Updating Phase 1 habitat map of Wales, UK, using satellite sensor data. ISPRS J. Photogramm. Remote Sens. 66(1), 81–102 (2011)

    Article  Google Scholar 

  26. Martínez, S., Ramil, P., Chuviecho, E.: Monitoring loss of biodiversity in cultural landscapes. New methodology based on satellite data. Landsc. Urban Plan. 94(2), 127–140 (2010)

    Article  Google Scholar 

  27. Moss, D., Wyatt, B.K.: The CORINE biotopes project: a database for conservation of nature and wildlife in the European community. Appl. Geogr. 14(4), 327–349 (1994)

    Article  Google Scholar 

  28. Moosmann, F., Nowak, E., Jurie, F.: Randomized Clustering Forests for Image Classification. IEEE Trans. Pattern Anal. Mach. Intell. 30(9), 1632–1646 (2008)

    Article  Google Scholar 

  29. Nilsback, M., Zisserman, A.: Automated Flower Classification. In Indian Conference on Computer Vision, Graphics and Image Processing (ICVGIP 2008), 722-729 (2008)

  30. Oliva, A., Torralba, A.: Modeling the shape of the scene: a holistic representation of the spatial envelope. Int. J. Comput. Vision 42, 145–175 (2001)

    Article  MATH  Google Scholar 

  31. Olsen, B.P.: Automatic change detection for validation of digital map databases. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. 34(B2), 569–574 (2004)

    Google Scholar 

  32. Park, A.J., Clare, J., Spicer, V., Brantingham, P.L., Calvert, T., Jenion, G.: Examining context-specific perceptions of risk: exploring the utility of “human-in- the-loop” simulation models for criminology. J. Exp. Criminol. 8, 29–47 (2012)

    Article  Google Scholar 

  33. Peters, J.De, Baets, B., Samson, R., Verhoest, N.E.C.: Modelling groundwater-dependent vegetation patterns using ensemble learning. Hydrol. Earth Syst. Sci. 12, 603–613 (2008)

    Article  Google Scholar 

  34. Peters, J.De, Baets, B., Samson, R., Verhoest, N.E.C., Degroeve, S., De Decker, P., Huybrechts, W.: Random forests as a tool for predictive ecohydrological modeling. Ecol. Model. 207, 304–318 (2007)

    Article  Google Scholar 

  35. Puri, M., Zhu, Z., Lubin, J., Pschar, T., Divakaran, A., Sawhney, H.S.: US Patent No. 8,439,683. U.S. Patent and Trademark Office, Washington, DC 2013

  36. Qiu, G.: Indexing chromatic and achromatic patterns for content-based colour image retrieval. Pattern Recogn. 35, 1675–1686, (2002)

  37. Rabinovich, A., Vedaldi, A., Galleguillos, C., Wiewiora, E., Belongie, S.: Objects in Context, IEEE International Conference on Computer Vision, 2007 (ICCV 2007), pp. 14–21, 11, Oct 2007

  38. Razeghi, O., Qiu, G., Williams, H., Thomas, K.: Computer Aided Skin Lesion Diagnosis with Humans in the Loop, Machine Learning in Medical Imaging, pp 266–274 (2012)

  39. Siddiquie, B., Feris, R.S., Davis, L.S.: Image Ranking and Retrieval Based on Multi-attribute Queries, IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2011), pp. 801–808, June 2011

  40. Scemske, D.W., Husband, B.C., Ruckelhaus, M.H., Goodwillie, C., Parker, I.M., Bishop, J.G.: Evaluating approaches to the conservation of rare and endangered plants. Ecology 75(3), 584–606 (1994)

    Article  Google Scholar 

  41. Shotton, J., Johnson, M., Cipolla, R: Semantic texton forests for image categorization and segmentation, IEEE Conference on Computer Vision and Pattern Recognition, 2008 (CVPR 2008), pp. 23–28, June 2008

  42. Smith, N.M., Lee, P.U., Prevôt, T., Mercer, J., Palmer III, E.A., Battiste, V., et al.: A human-in-the-loop evaluation of air-ground trajectory negotiation. Am. Inst. Aeronaut. Astronaut., Reston (2004)

    Book  Google Scholar 

  43. Thompson, M.: A standard land-cover classification scheme for remote-sensing applications in South Africa. S. Afr. J. Sci. 92(1), 34–42 (1996)

    Google Scholar 

  44. Torres, M.: Automatic Habitat Classification Using Aerial Imagery, GIS Research UK 20th Annual Conference (GISRUK 2012). Lancaster, UK, April 2012

  45. Torres, M., Qiu, G.: Automatic Habitat Classification using Image Analysis and Random Forest, Ecological Informatics. Available online 2 September 2013

  46. Torres, M., Qiu, G. 2012. Grass, scrub, trees and random forest. Proceedings of the 1st ACM International Workshop on Multimedia Analysis for Ecological Data, pp. 1–6. Nara, Japan, November 2012

  47. UK Biodiversity Steering Group: Biodiversity: The UK Steering Group Report, Vol 1: Meeting the Rio challenge. London, HMSO (1995)

  48. van Kooten, G.C., Stennes, B., Krcmar, E., van Gorkom, R.: Economics of afforestation for carbon sequestration in western Canada. For Chron 76, 165–172 (2000)

    Article  Google Scholar 

  49. Vedaldi, A., Fulkerson, B.: Vlfeat: An open and portable library of computer vision algorithms. http://www.vlfeat.org/ (2008). Accessed Apr 2014

  50. Vedaldi, A., Gulshan, V., Varma, M., Zisserman, A.: Multiple Kernels for Object Detection. In Proceedings of the International Conference on Computer Vision (ICCV 2009)

  51. Wah, C.,Branson, S., Perona, P., Belongie, S.: Multiclass Recognition and Part Localization with Humans in the Loop. In: ICCV 2011

  52. Wang, J., Markert, K., Everingham, M.: Learning Models for Object Recognition from Natural Language Descriptions. In: Proceedings of the 20th British Machine Vision Conference (BMVC2009)

  53. Zhou, N., Cheung, W.K., Qiu, G., Xue, X.: A Hybrid Probabilistic Model for Unified Collaborative and Content-Based Image Tagging. IEEE Trans. Pattern Anal. Mach. Intell. 33(7), 1281–1294 (2011)

    Article  Google Scholar 

  54. Zhu, F., Bosch, M., Khanna, N., Boushey, C.J., Delp, E.J.: Multilevel Segmentation for Food Classification in Dietary Assessment, 7th International Symposium on Image and Signal Processing and Analysis (ISPA 2011), pp. 337–342

Download references

Acknowledgments

Mercedes Torres is supported by the Horizon Doctoral Training Centre at the University of Nottingham (RCUK Grant No. EP/G037574/1). Guoping Qiu is partially supported by the International Doctoral Innovation Centre (IDIC) at the University of Nottingham Ningbo China (UNNC) and by Ningbo Science and Technology Bureau projects 2013D10008 and 2012B10055.

Author information

Authors and Affiliations

  1. University of Nottingham, Jubilee Campus, Wollaton Road, Nottingham, NG8 1BB, UK

    Mercedes Torres & Guoping Qiu

  2. The University of Nottingham, Ningbo, China

    Guoping Qiu

Authors
  1. Mercedes Torres
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Guoping Qiu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mercedes Torres.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Torres, M., Qiu, G. Habitat image annotation with low-level features, medium-level knowledge and location information. Multimedia Systems 22, 767–782 (2016). https://doi.org/10.1007/s00530-014-0445-2

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00530-014-0445-2

Keywords

Search

Navigation