Abstract
The ability for researchers to re-identify animal individuals upon re-encounter is fundamental for the study of population dynamics, community, and behavioural ecology. Animal re-identification is traditionally performed using tagging or DNA sampling, which is laborious, invasive to an animal, and expensive. An alternative approach to re-identify is the use of computer vision in combination with pattern recognition algorithms. Deep learning has accelerated the success when solving pattern recognition in the field of computer vision when high data volume is available; however, conventional deep learning approaches require ample training data for a fixed number of classes. An alternative deep learning paradigm is similarity comparison networks which are trained to identify if two inputs are the same or different. This principle can be applied to images of animal individuals and allows for the re-identification of individuals beyond the original training data. Here, we test the potential and generality of similarity comparison networks for animal re-identification considering five datasets of different species: humans, chimpanzees, humpback whales, fruit flies, and Siberian tigers, each with their own unique set of challenges. We compare 10 similarity comparison networks by testing five well-established network architectures (AlexNet, VGG19, DenseNet201, ResNet152, and InceptionV3) and two different methods to train each of them: contrastive and triplet loss. Models were trained to re-identify individuals and those trained using the triplet loss outperformed contrastive loss for all species. Our work shows that without any species-specific modifications, similarity comparison networks can act as a general purpose animal re-identification system considering individuals from images. Our expectation is that similarity comparison networks are the beginning of a major trend that has the potential to revolutionize the study of population dynamics, community, and behavioural ecology. This work is an extension of a technical report presented at WACV 2020 (IEEE/CVF winter conference on applications of computer vision workshops) catered for an ecological audience.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Change history
01 October 2022
Supplementary Information was updated.
References
Arzoumanian Z, Holmberg J, Norman B (2005) An astronomical pattern-matching algorithm for computer-aided identification of whale sharks Rhincodon typus. J Appl Ecol 42(6):999–1011. https://doi.org/10.1111/j.1365-2664.2005.01117.x
Beery S, Wu G, Rathod V, Votel R, Huang J (2020) Context R-CNN: long term temporal context for per-camera object detection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 13075–13085. https://doi.org/10.1109/CVPR42600.2020.01309
Bergamini L, Porrello A, Dondona AC, Del Negro E, Mattioli M, D’alterio N, Calderara S (2018) Multi-views embedding for cattle re-identification. In: 2018 14th international conference on signal-image technology & internet-based systems (SITIS), pp 184–191. https://arxiv.org/abs/1902.04886
Bianco S, Cadene R, Celona L, Napoletano P (2018) Benchmark analysis of representative deep neural network architectures. IEEE Access 6:64270–64277. https://doi.org/10.1109/ACCESS.2018.2877890
Bouma S, Pawley MDM, Hupman K, Gilman A (2018) Individual common dolphin identification via metric embedding learning. In: 2018 International conference on image and vision computing New Zealand (IVCNZ), pp 1–6. https://doi.org/10.1109/IVCNZ.2018.8634778
Bromley J, Guyon I, LeCun Y, Sa¨ckinger E, Shah R (1994) Signature verification using a ‘siamese’ time delay neural network. In: Advances in neural information processing systems, pp 737–744. https://proceedings.neurips.cc/paper/1993/file/288cc0ff022877bd3df94bc9360b9c5d-Paper.pdf
Brust C, Burghardt T, Groenenberg M, Ka¨ding C, Ku¨hl HS, Manguette ML, Denzler J (2017) Towards automated visual monitoring of individual gorillas in the wild. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2820–2830. https://doi.org/10.1109/ICCVW.2017.333
Carter SJB, Bell IP, Miller JJ, Gash PP (2014) Automated marine turtle photograph identification using artificial neural networks, with application to green turtles. J Exp Mar Biol Ecol 452:105–110. https://doi.org/10.1016/j.jembe.2013.12.010
Krebs CJ (1989) Ecological methodology. Technical report, Harper & Row, New York. https://openlibrary.org/books/OL2043033M/Ecological_methodology
Chechik G, Sharma V, Shalit U, Bengio S (2010) Large scale online learning of image similarity through ranking 11:1109–1135. https://jmlr.org/papers/v11/chechik10a.html
Cheeseman T, Southerland K, Park J, Olio M, Flynn K, Calambokidis J, Jones L, Garrigue C, Frisch A, Howard A, Reade W, Neilson J, Gabriele C, Clapham P (2022) Advanced image recognition: a fully automated, high-accuracy photo-identification matching system for humpback whales. Mamm Biol (special Issue). https://doi.org/10.1007/s42991-021-00180-9
Chelak I, Nepovinnykh E, Eerola T, Kalviainen H, Belykh I (2021) Identification of Saimaa ringed seal individuals using transfer learning. In: International conference on advanced concepts for intelligent vision systems. LNCS, vol 11182, pp 211–222. https://doi.org/10.1007/978-3-030-01449-0_18
Chen P, Swarup P, Matkowski WM, Kong AW, Han S, Zhang Z, Rong H (2020) A study on giant panda recognition based on images of a large proportion of captive pandas. Ecol Evol 10(7):3561–3573. https://doi.org/10.1002/ece3.6152
Clapham M, Miller E, Nguyen M, Darimont CT (2020) Automated facial recognition for wildlife that lack unique markings: a deep learning approach for brown bears. Ecol Evol 10(23):12883–12892. https://doi.org/10.1002/ece3.6840
Deb D, Wiper S, Russo A, Gong S, Shi Y, Tymoszek C, Jain A (2018) Face recognition: primates in the wild. In: IEEE 9th international conference on biometrics theory, applications and systems (BTAS). 18619856. https://doi.org/10.1109/BTAS.2018.8698538
Dlamini N, van Zyl TL (2020) Automated identification of individuals in wildlife population using siamese neural networks. In: 2020 7th international conference on soft computing & machine intelligence (ISCMI), pp 224–228. https://doi.org/10.1109/ISCMI51676.2020.9311574
Dlamini N, van Zyl TL (2021) Comparing class-aware and pairwise loss functions for deep metric learning in wildlife re-identification. Sensors 21(18):6109. https://doi.org/10.3390/s21186109
Ferreira AC, Silva LR, Renna F, Brandl HB, Renoult JP, Farine DR, Covas R, Doutrelant C (2020) Deep learning-based methods for individual recognition in small birds. Methods Ecol Evol 11(9):1072–1085. https://doi.org/10.1111/2041-210X.13436
Foster RJ, Harmsen BJ (2012) A critique of density estimation from camera-trap data. J Wildl Manag 76(2):224–236. https://doi.org/10.1002/jwmg.275
Freytag A, Rodner E, Simon M, Loos A, Kühl HS, Denzler J (2016) Chimpanzee faces in the wild: log-euclidean CNNs for predicting identities and attributes of primates. In: German conference on pattern recognition, pp 51–63. https://doi.org/10.1007/978-3-319-45886-1_5
Fukushima K (1980) Neocognitron: a self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biol Cybernetics 36:193–202. https://doi.org/10.1007/BF00344251
Haurum JB, Karpova A, Pedersen M, Bengtson SH, Moeslund TB (2020) Re-identification of zebrafish using metric learning. In: Proceedings of the IEEE/CVF winter conference on applications of computer vision workshops, pp 1–11. https://doi.org/10.1109/WACVW50321.2020.9096922
He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778. https://doi.org/10.1109/CVPR.2016.90
He K, Gkioxari G, Dolla´r P, Girshick R (2017) Mask r-cnn. In: Proceedings of the IEEE international conference on computer vision, pp 2961–2969. https://doi.org/10.1109/ICCV.2017.322
Hearst MA, Dumais ST, Osuna E, Platt J, Scholkopf B (1998) Support vector machines. IEEE Intell Syst Their Appl 13(4):18–28. https://doi.org/10.1109/5254.708428
Hiby L, Lovell P (1990) Computer aided matching of natural markings: a prototype system for grey seals. Rep Int Whal Comm 12:57–61.
Hinton GE, Osindero S, Teh Y (2006) A fast learning algorithm for deep belief nets. Neural Comput 18(7):1527–1554. https://doi.org/10.1162/neco.2006.18.7.1527
Hoffer E, Ailon N (2015) Deep metric learning using triplet network. In: International workshop on similarity-based pattern recognition, pp 84–92. https://doi.org/10.1007/978-3-319-24261-3_7
Holzinger A (2016) Interactive machine learning for health informatics: when do we need the human-in-the-loop? Brain Inform 3(2):119–131. https://doi.org/10.1007/s40708-016-0042-6
Hosmer DW Jr, Lemeshow S, Sturdivant RX (2013) Applied logistic regression, vol 398. Wiley. https://doi.org/10.1002/9781118548387
Huang G, Liu Z, Van Der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4700–4708. https://doi.org/10.1109/CVPR.2017.243
Humpback whale identification challenge (2018) https://www.kaggle.com/c/whale-categorization-playground. Accessed 15 May 2018
iWildcam 2018 camera trap challenge (2018) https://www.kaggle.com/c/iwildcam2018. Accessed 11 July 2018
Jaderberg M, Simonyan K, Zisserman A (2015) Spatial transformer networks. In: Advances in neural information processing systems, pp 2017–2025. https://arxiv.org/abs/1506.02025
Kabuga E (2019) Using neural networks to identify individual animals from photographs. Master’s thesis, Faculty of Science. https://open.uct.ac.za/handle/11427/31834
Keller JM, Gray MR, Givens JA (1985) A fuzzy k-nearest neighbor algorithm. IEEE Trans Syst Man Cybern 4:580–585. https://scirp.org/reference/referencespapers.aspx?referenceid=29774
Kingma D, Ba J (2014) Adam: A method for stochastic optimization. https://arxiv.org/abs/1412.6980
Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1097–1105. https://doi.org/10.1145/3065386
Kulits P, Wall J, Bedetti A, Henley M, Beery S (2021) Elephantbook: a semi-automated human-in-the-loop system for elephant re-identification. In: ACM SIGCAS conference on computing and sustainable societies (COMPASS), pp 88–98. https://doi.org/10.1145/3460112.3471947
LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(7553):436–444. https://doi.org/10.1038/nature14539
Li S, Li J, Lin W, Tang H (2019) Amur tiger re-identification in the wild. https://arxiv.org/abs/1906.05586
Lisanti G, Masi I, Bagdanov AD, Del Bimbo A (2015) Person re-identification by iterative re-weighted sparse ranking. IEEE Trans Pattern Anal Mach Intell 37(8):1629–1642. https://doi.org/10.1109/TPAMI.2014.2369055
Loos A, Ernst A (2013) An automated chimpanzee identification system using face detection and recognition. EURASIP J Image and Video Proc 2013(1):49. https://doi.org/10.1186/1687-5281-2013-49
Martinel N, Das A, Micheloni C, Roy-Chowdhury AK (2015) Re-identification in the function space of feature warps. IEEE Trans Pattern Anal Mach Intell 37(8):1656–1669. https://doi.org/10.1109/TPAMI.2014.2377748
Meek PD, Vernes K, Falzon G (2013) On the reliability of expert identification of small-medium sized mammals from camera trap photos. Wildl Biol Pract 9(2):1–19. https://hdl.handle.net/1959.11/13729
Miele V, Dussert G, Spataro B, Chamaille´-Jammes S, Allaine´ D, Bonenfant C (2021) Revisiting animal photo-identification using deep metric learning and network analysis. Methods Ecol Evol 12(5):863–873. https://doi.org/10.1111/2041-210X.13577
Nepovinnykh E, Eerola T, Kalviainen H (2020) Siamese network based pelage pattern matching for ringed seal re-identification. In: Proceedings of the IEEE/CVF winter conference on applications of computer vision workshops, pp 25–34. https://doi.org/10.1109/WACVW50321.2020.9096935
Ng H, Winkler S (2014) A data-driven approach to cleaning large face datasets. In: 2014 IEEE international conference on image processing (ICIP), pp 343–347. https://doi.org/10.1109/ICIP.2014.7025068
Norouzzadeh MS, Nguyen A, Kosmala M, Swanson A, Palmer M, Packer C, Clune J (2017) Automatically identifying, counting, and describing wild animals in camera-trap images with deep learning. Proc Natl Acad Sci 115(25):E5716–E5725. https://doi.org/10.1073/pnas.1719367115
Norouzzadeh MS, Morris D, Beery S, Joshi N, Jojic N, Clune J (2021) A deep active learning system for species identification and counting in camera trap images. Methods Ecol Evol 12(1):150–161. https://doi.org/10.1111/2041-210X.13504
Pal M (2005) Random forest classifier for remote sensing classification. Int J Remote Sens 26(1):217–222. https://doi.org/10.1080/01431160412331269698
Parham J, Stewart C, Crall J, Rubenstein D, Holmberg J, Berger-Wolf T (2018) An animal detection pipeline for identification. In: 2018 IEEE winter conference on applications of computer vision (WACV), pp 1075–1083. https://doi.org/10.1109/WACV.2018.00123
Redmon J, Divvala S, Girshick R, Farhadi A (2016) You only look once: Unified, real-time object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 779–788. https://doi.org/10.1109/CVPR.2016.91
Ren S, He K, Girshick R, Sun J (2017) Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Trans Pattern Anal Mach Intell 39(6):1137–1149. https://doi.org/10.1109/TPAMI.2016.2577031
Schneider J, Murali N, Taylor GW, Levine JD (2018a) Can drosophila melanogaster tell whos who? PLoS ONE 13(10):e0205043. https://doi.org/10.1371/journal.pone.0205043
Schneider S, Taylor GW, Kremer SC (2018b) Deep learning object detection methods for ecological camera trap data. In: 2018 15th Conference on computer and robot vision (CRV), pp 321–328. https://doi.org/10.1109/CRV.2018.00052
Schneider S, Taylor GW, Linquist S, Kremer SC (2019) Past, present and future approaches using computer vision for animal re-identification from camera trap data. Methods Ecol Evol 10(4):461–470. https://doi.org/10.1111/2041-210X.13133
Schneider S, Greenberg S, Taylor GW, Kremer SC (2020a) Three critical factors affecting automated image species recognition performance for camera traps. Ecol Evol 10(7):3503–3517. https://doi.org/10.1002/ece3.6147
Schneider S, Taylor GW, Kremer SC (2020b) Similarity learning networks for animal individual re- identification-beyond the capabilities of a human observer. In: Proceedings of the IEEE/CVF winter conference on applications of computer vision workshops, pp 44–52. https://doi.org/10.1109/WACVW50321.2020.9096925
Schofield D, Nagrani A, Zisserman A, Hayashi M, Matsuzawa T, Biro D, Carvalho S (2019) Chimpanzee face recognition from videos in the wild using deep learning. Sci Adv 5(9):eaaw0736. https://doi.org/10.1126/sciadv.aaw0736
Schroff F, Kalenichenko D, Philbin J (2015) Facenet: a unified embedding for face recognition and clustering. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 815– 823. https://doi.org/10.1109/CVPR.2015.7298682
Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. https://arxiv.org/abs/1409.1556
Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1–9. https://doi.org/10.1109/CVPR.2015.7298594
Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z (2016) Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2818–2826. https://doi.org/10.1109/CVPR.2016.308
Torchvision. https://pytorch.org/docs/stable/torchvision/index.html.
Whitehead H (1990) Computer assisted individual identification of sperm whale flukes. Rep Int Whal Comm 12:71–77.
Wolf L, Hassner T, Maoz I (2011) Face recognition in unconstrained videos with matched background similarity. In CVPR 2011:529–534. https://doi.org/10.1109/CVPR.2011.5995566
Zheng L, Shend L, Tian L, Wang S, Wang J, Tian Q (2015) Scalable person re-identification: A benchmark. In: Proceedings of the IEEE international conference on computer vision, pp 1116–1124. https://doi.org/10.1007/s11263-019-01274-1
Zhu XJ (2005) Semi-supervised learning literature survey. University of Wisconsis-Madison Department of Computer Sciences. http://digital.library.wisc.edu/1793/60444
Author information
Authors and Affiliations
Contributions
SS was primary motivator of this work, responsible for the writing, experimental design, training and implementation of the models, and networking between authors. GT and SK assisted in conceptualizing the deep learning components of the work. All authors were responsible for revising the initial manuscript drafted by Stefan Schneider.
Corresponding author
Additional information
Handling editors: Leszek Karczmarski and Scott Y.S. Chui.
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
This article is a contribution to the special issue on “Individual Identification and Photographic Techniques in Mammalian Ecological and Behavioural Research – Part 1: Methods and Concepts” — Editors: Leszek Karczmarski, Stephen C.Y. Chan, Daniel I. Rubenstein, Scott Y.S. Chui and Elissa Z. Cameron.
Supplementary Information
Below is the link to the electronic supplementary material.
Rights and permissions
Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Schneider, S., Taylor, G.W. & Kremer, S.C. Similarity learning networks for animal individual re-identification: an ecological perspective. Mamm Biol 102, 899–914 (2022). https://doi.org/10.1007/s42991-021-00215-1
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s42991-021-00215-1