Abstract
We describe the development and application of a new convolutional neural network-based photo-identification algorithm for individual humpback whales (Megaptera novaeangliae). The method uses a Densely Connected Convolutional Network (DenseNet) to extract special keypoints of an image of the ventral surface of the fluke and then a separate DenseNet trained to look for features within these keypoints. The extracted features are then compared against those of the reference set of previously known humpback whales for similarity. This offers the potential to successfully automate recognition of individuals in large photographic datasets such as in ocean basin-wide marine mammal studies. The algorithm requires minimal image pre-processing and is capable of accurate, rapid matching of fair to high-quality humpback fluke photographs. In real world testing compared to manual image matching, the algorithm reduces image management time by at least 98% and reduces error rates of missing potential matches from approximately 6–9% to 1–3%. The success of this new system permits automated comparisons to be made for the first time across photo-identification datasets with tens to hundreds of thousands of individually identified encounters, with profound implications for long-term and large population studies of the species.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Change history
01 October 2022
Supplementary Informaiton was updated.
Notes
References
Acevedo J, Félix F (2017) Report of the Workshop on Southern Hemisphere blue, fin and humpback whale photo-identification catalogues from the Central and Eastern South Pacific and the Antarctic Peninsula. Rep Int Whal Commun
Arzoumanian Z, Holmberg J, Norman B (2005) An astronomical pattern-matching algorithm for computer-aided identification of whale sharks Rhincodon typus. J Appl Ecol 42:999–1011. https://doi.org/10.1111/j.1365-2664.2005.01117.x
Barlow J, Calambokidis J, Falcone EA, Baker CS, Burdin AM, Clapham PJ, Ford JKB, Gabriele CM, LeDuc R, Mattila DK, Quinn TJ, Rojas-Bracho L, Straley JM, Taylor BL, Urbán RJ, Wade P, Weller D, Witteveen BH, Yamaguchi M (2011) Humpback whale abundance in the North Pacific estimated by photographic capture-recapture with bias correction from simulation studies. Mar Mamm Sci 27:793–818. https://doi.org/10.1111/j.1748-7692.2010.00444.x
Bettridge S, Baker CS, Barlow J, Clapham PJ, Ford M, Gouveia D, Mattila DK, Pace RMI, Rosel PE, Silber GK, Wade PR (2015) Status review of the humpback whale (Megaptera novaeangliae) under the endangered species act. NOAA-TM-NMFS-SWFSC- 540. National Marine Fisheries Service, Silver Spring, MD. p 240
Blackmer AL, Anderson SK, Weinrich MT (2000) Temporal variability in features used to photo-identify humpback whales (Megaptera novaeangliae). Mar Mamm Sci 16:338–354. https://doi.org/10.1111/j.1748-7692.2000.tb00929.x
Bogucki R, Cygan M, Khan CB, Klimek M, Milczek JK, Mucha M (2019) Applying deep learning to right whale photo identification. Conserv Biol 33:676–684. https://doi.org/10.1111/cobi.13226
Calambokidis J, Falcone EA, Quinn TJ, Burdin AM, Clapham PJ, Ford JKB, Gabriele CM, LeDuc R, Mattila D, Bracho LR-, Janice M. Straley, Taylor BL, R. JU, Weller D, Witteveen BH, Yamaguchi M, Bendlin A, Camacho D, Flynn K, Havron A, Huggins J, Maloney N (2008) SPLASH: Structure of populations, levels of abundance and status of humpback whales in the North Pacific. Final report for Contract AB133F-03-RP-00078, available from Cascadia Research, Olympia, WA
Clapham M, Miller E, Nguyen M, Van Horn RC (2022) Multispecies facial detection for individual identification of wildlife: a case study across ursids. Mamm Biol (Special Issue) 102(3). https://doi.org/10.1007/s42991-021-00168-5
Clutton-Brock TH, Sheldon BC (2010) Individuals and populations: the role of long-term, individual-based studies of animals in ecology and evolutionary biology. Trends Ecol Evol 25:562–573. https://doi.org/10.1016/j.tree.2010.08.002
Crall JP, Stewart C V, Berger-Wolf TY, Rubenstein DI, Sundaresan SR (2013) HotSpotter—patterned species instance recognition. In: 2013 IEEE workshop on applications of computer vision (WACV), pp 230–237
Deng J, Deng J, Dong W, Socher R, Li L, Li K, Fei-fei L (2009) Imagenet: a large-scale hierarchical image database. 2009 IEEE conference on computer vision and pattern recognition
Deng J, Guo J, Xue N, Zafeiriou S (2018) ArcFace: Additive Angular Margin Loss for Deep Face Recognition. In: Proc IEEE comput soc conf comput vis pattern recognit 2019-June, pp 4685–4694
Fleming A, Jackson J (2011) Global review of humpback whales (Megaptera novaeangliae). NOAA technical memorandum. U.S. Department of Commerice, National Oceanic and Atmospheric Administration, National Marine Fisheries Service
Franklin T, Franklin W, Brooks L, Harrison P, Burns D, Holmberg J, Calambokidis J (2020) Photo-identification of individual humpback whales (Megaptera novaeangliae) using all available natural marks: Implications for misidentification and automated algorithm matching technology. J Cetacean Res Manag 21:71–83. https://doi.org/10.47536/JCRM.V21I1.186
Friday N, Smith TD, Stevick PT, Allen J (2000) Measurement of photographic quality and individual distinctiveness for the photographic identification of humpback whales, Megaptera novaeangliae. Mar Mamm Sci 16:355–374. https://doi.org/10.1111/j.1748-7692.2000.tb00930.x
Garrigue C, Franklin T, Constatine R, Russell K, Burns D, Poole M, Paton D, Hauser N, Oremus M, Mattila D, Gibbs N, Franklin W, Robbins J, Clapham P, Baker C (2011) First assessment of interchange of humpback whales between Oceania and the East coast of Australia. J Cetacean Res Manag Special Issue 3:269–274
Hammond PS (1986) Estimating the size of naturally marked whale populations using capture-recapture techniques. Rep Int Whal Commun 8:253–282
Hammond PS, Mizroch SA, Donovan GP (1990) Individual recognition of cetaceans: use of photo-identification and other techniques to estimate population parameters: incorporating the proceedings of the symposium and workshop on individual recognition and the estimation of cetacean population parame. International Whaling Commission. Special Issue 12:448
Huang G, Liu Z, van der Maaten L, Weinberger KQ (2016) Densely connected convolutional networks. In: Proc—30th IEEE Conf Comput Vis Pattern Recognition, CVPR 2017 2017-January, pp 2261–2269
Jurasz C, Jurasz V (1978) Results of 1977 Studies on Humpback Whales in Glacier Bay National Monument. Final Rept.
Karczmarski L, Chan SCY, Rubenstein DI, Chui SYS, Cameron EZ (2022a) Individual identification and photographic techniques in mammalian ecological and behavioural research – Part 1: Methods and concepts. Mamm Biol (Special Issue) 102(3). https://link.springer.com/journal/42991/volumes-and-issues/102-3
Karczmarski L, Chan SCY, Chui SYS, Cameron EZ (2022b) Individual identification and photographic techniques in mammalian ecological and behavioural research – Part 2: Field studies and applications. Mamm Biol (Special Issue) 102(4). https://link.springer.com/journal/42991/volumes-and-issues/102-4
Katona SK, Whitehead HP (1981) Identifying Humpback Whales using their natural markings. Polar Rec (gr Brit) 20:439–444. https://doi.org/10.1017/S003224740000365X
Katona S, Baxter B, Brazier O, Kraus S, Perkins J, Whitehead H (1979) Identification of Humpback Whales by Fluke Photographs. Behavior of marine animals. Springer, New York, pp 33–44
Khan C, Blount D, Parham J, Holmberg J, Hamilton P, Charlton C, Christiansen F, Johnston D, Rayment W, Dawson S, Vermeulen E, Rowntree V, Groch K, Levenson JJ, Bogucki R (2022) Artificial intelligence for right whale photo identification: from data science competition to worldwide collaboration. Mamm Biol (Special Issue) 102(3). https://doi.org/10.1007/s42991-022-00253-3
Kniest E, Burns D, Harrison P (2010) Fluke Matcher: a computer-aided matching system for humpback whale (Megaptera novaeangliae) flukes. Mar Mamm Sci 26:744–756. https://doi.org/10.1111/j.1748-7692.2009.00368.x
Langley I, Hague E, Arso Civil M (2022) Assessing the performance of open-source, semi-automated pattern recognition software for harbour seal (P. v. vitulina) photo ID. Mamm Biol (Special Issue) 102(3). https://doi.org/10.1007/s42991-021-00165-8
Link WA (2003) Nonidentifiability of population size from capture-recapture data with heterogeneous detection probabilities. Biometrics 59:1123–1130
Mizroch SA, Beard JA, Lynde M (1990) Computer assisted photo-identification of humpback whales. Rep Int Whal Commun 12:63–70
Schevill WE, Backus RH (1960) Daily Patrol of a Megaptera. J Mamm 41:279. https://doi.org/10.2307/1376380
Schneider S, Taylor GW, Linquist S, Kremer SC (2019) Past, present and future approaches using computer vision for animal re-identification from camera trap data. Methods Ecol Evol 10:461–470. https://doi.org/10.1111/2041-210X.13133
Smith TD, Allen J, Clapham PJ, Hammond PS, Katona S, Larsen F, Lien J, Mattila D, Palsboll PJ, Sigurjonsson J, Stevick PT, Oien N (1999) An ocean-basin-wide Mark-Recapture Study of the North Atlantic Humpback Whale (Megaptera novaeangliae). Mar Mamm Sci 15:1–32. https://doi.org/10.1111/j.1748-7692.1999.tb00779.x
Stevick P, Allen J, Clapham P, Friday N, Katona S, Larsen F, Lien J, Mattila D, Palsbøll P, Sigurjónsson J, Smith T, Øien N, Hammond P (2003) North Atlantic humpback whale abundance and rate of increase four decades after protection from whaling. Mar Ecol Prog Ser 258:263–273. https://doi.org/10.3354/meps258263
Sullivan BL, Wood CL, Iliff MJ, Bonney RE, Fink D, Kelling S (2009) eBird: a citizen-based bird observation network in the biological sciences. Biol Conserv 142:2282–2292. https://doi.org/10.1016/j.biocon.2009.05.006
Taigman Y, Yang M, Ranzato M, Wolf L (2014) DeepFace: closing the gap to human-level performance in face verification. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. IEEE Computer Society, pp 1701–1708
Weideman HJ, Jablons ZM, Holmberg J, Flynn K, Calambokidis J, Tyson RB, Allen JB, Wells RS, Hupman K, Urian K, Stewart CV (2017) Integral Curvature Representation and Matching Algorithms for Identification of Dolphins and Whales. ICCV 2017 Workshop on Visual Wildlife Monitoring. arXiv:1708.07785v1
Acknowledgements
We extend special thanks and acknowledgement to all data contributors; a full list of 970 image contributors other than the authors, is given as Supplementary Information. This includes all contributors for whom we have complete attribution information whose images were included in the algorithm development competition dataset. The competition dataset would not have been possible to create without efforts from Deana Glenz, Kate Spencer, Denny Zwiefelhofer, Jenny Grayson, Lucy Payne and Tory Johnson. This publication uses data generated via the Zooniverse.org platform, development of which is funded by generous support, including a Global Impact Award from Google, and by a grant from the Alfred P. Sloan Foundation. The Zooniverse data processing would not have been successful without help from Peter Mason. We also thank Jan Straley, Dan Burns, Tom Hart, Leszek Karczmarski and anonymous peer reviewers for substantial manuscript contributions.
Funding
Funding for this research was provided by a grant from Cheesemans’ Ecology Safaris, San Jose, California (www.cheesemans.com).
Author information
Authors and Affiliations
Contributions
TC conceived the study, secured funding, collected and managed data, oversaw algorithm development, analyzed results and led manuscript preparation, KS directed and executed information architecture coding and contributed to manuscript preparation. JP authored the implemented Kaggle competition algorithm, MO managed data, KF and JC contributed to study conception and contributed image and identification data, LJ contributed image and identification data, directed testing manual versus automated matching and contributed to manuscript preparation, CG, JN and CG contributed image and identification data and contributed to manuscript preparation, AF contributed image and identification data, AH and WR directed the Kaggle competition, and PC contributed to study conception and contributed to manuscript preparation.
Corresponding author
Ethics declarations
Conflict of interest
Not applicable.
Ethics approval
Not applicable.
Consent to participate
Not applicable.
Consent for publication
Not applicable.
Availability of data and materials (data transparency)
Most of the photo-ID data utilized in this research is visible via www.happywhale.com, including individual sighting histories, available for use within contributor-set licensing as specified with each image. Images used for the Kaggle competition are available via https://www.kaggle.com/c/humpback-whale-identification.
Code availability (software application or custom code)
Open source repositories for the top five scoring algorithms, as posted at the completion of the Kaggle competition, are available via https://www.kaggle.com/c/humpback-whale-identification/discussion. Access to the implemented algorithm and supporting information architecture is available for use at no cost via the web platform www.happywhale.com and via the authors.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Handling editors: Leszek Karczmarski and Stephen C.Y. Chan.
This article is a contribution to the special issue on “Individual Identification and Photographic Techniques in Mammalian Ecological and Behavioural Research - Part 1: Methods and Concepts” – Editors: Leszek Karczmarski, Stephen C.Y. Chan, Daniel I. Rubenstein, Scott Y.S. Chui and Elissa Z. Cameron.
Supplementary Informaiton was updated.
Supplementary Information
Below is the link to the electronic supplementary material.
Rights and permissions
Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Cheeseman, T., Southerland, K., Park, J. et al. Advanced image recognition: a fully automated, high-accuracy photo-identification matching system for humpback whales. Mamm Biol 102, 915–929 (2022). https://doi.org/10.1007/s42991-021-00180-9
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s42991-021-00180-9