Abstract
High-throughput phenotyping is now central to the progress of plant sciences, accelerated breeding, and precision farming. The power of phenotyping comes from the automated, rapid, non-invasive collection of large datasets describing plant objects. In this context, the goal of extracting relevant information from different kinds of images is of paramount importance. We review both the spectral and machine learning-based approaches to imaging of plants for the purpose of their phenotyping. The advantages and drawbacks of both approaches will be discussed with a focus on the monitoring of plants. We argue that an emerging approach combining the strengths of the spectral and the machine learning-based approaches will remain a promising direction in plant phenotyping in the nearest future.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data Availability
The data are available from the corresponding author on reasonable request.
References
Al Riza DF, Rulin C, Tun NTT, Yi PPL, Thwe AA, Myint KT, Kondo N (2023) Mango (Mangifera indica cv. Sein Ta Lone) ripeness level prediction using color and textural features of combined reflectance-fluorescence images. J Agric Food Res 11:100477. https://doi.org/10.1016/j.jafr.2022.100477
Alom MZ et al. (2018) The history began from alexnet: a comprehensive survey on deep learning approaches arXiv preprint arXiv:180301164. https://doi.org/10.48550/arXiv.1803.01164
Arya S, Sandhu KS, Singh J, Kumar S (2022) Deep learning: as the new frontier in high-throughput plant phenotyping Euphytica:218. https://doi.org/10.1007/s10681-022-02992-3
Bargoti S, Underwood J (2017) Deep fruit detection in orchards. In: 2017 IEEE international conference on robotics and automation (ICRA). IEEE, pp 3626–3633. https://doi.org/10.1109/ICRA.2017.7989417
Chen Y, Zhu L, Ghamisi P, Jia X, Li G, Tang L (2017) Hyperspectral images classification with Gabor filtering and convolutional neural network. IEEE Geosci Remote Sens Lett 14:2355–2359. https://doi.org/10.1109/LGRS.2017.2764915
Cross GR, Jain AK (1983) Markov random field texture models IEEE Transactions on Pattern Analysis and Machine Intelligence 25-39. https://doi.org/10.1109/TPAMI.1983.4767341
Davies KM et al (2022) Evolution and function of red pigmentation in land plants. Ann Bot. https://doi.org/10.1093/aob/mcac109
Dosovitskiy A et al (2020) An image is worth 16x16 words: transformers for image recognition at scale arXiv preprint arXiv:201011929. https://doi.org/10.48550/arXiv.2010.11929
Egmont-Petersen M, de Ridder D, Handels H (2002) Pattern Recognit Lett 35:2279–2301. https://doi.org/10.1016/S0031-3203(01)00178-9
Fauvel M, Benediktsson JA, Chanussot J, Sveinsson JR (2008) Spectral and spatial classification of hyperspectral data using SVMs and morphological profiles. IEEE Trans Geosci Remote Sens 46:3804–3814. https://doi.org/10.1109/TGRS.2008.922034
Féret JB, Gitelson AA, Noble SD, Jacquemoud S (2017) PROSPECT-D: towards modeling leaf optical properties through a complete lifecycle Remote Sensing of. Environment 193:204–215. https://doi.org/10.1016/j.rse.2017.03.004
Frank S, Bugliarello E, Elliott D (2021) Vision-and-language or vision-for-language? On cross-modal influence in multimodal transformers arXiv preprint arXiv:210904448. https://doi.org/10.48550/arXiv.2109.04448
Gabor D (1946) Theory of communication. Part 1: the analysis of information. J Inst Elect Eng-part III: Radio Commun Eng 93:429–441. https://doi.org/10.1049/ji-3-2.1946.0074
Gao S, Xu J-h (2022) Hyperspectral image information fusion-based detection of soluble solids content in red globe grapes. Comput Electron Agric 196:106822. https://doi.org/10.1016/j.compag.2022.106822
Gené Mola J, Sanz Cortiella R, Rosell Polo JR, Morros Rubió JR, Ruiz Hidalgo J, Vilaplana Besler V, Gregorio López E (2020) Fuji-SfM dataset. https://doi.org/10.5281/zenodo.3712808
Girshick R, Donahue J, Darrell T (2014) Malik J Rich feature hierarchies for accurate object detection and semantic segmentation. Proceedings of the IEEE conference on computer vision and pattern recognition, In, pp 580–587
Gitelson A, Arkebauer T, Viña A, Skakun S, Inoue Y (2021) Evaluating plant photosynthetic traits via absorption coefficient in the photosynthetically active radiation region. Remote Sens Environ:258. https://doi.org/10.1016/j.rse.2021.112401
Gitelson A, Solovchenko A (2017) Generic algorithms for estimating foliar pigment content. Geophys Res Lett 44:9293–9298. https://doi.org/10.1029/2006GL026457
Gitelson A, Solovchenko A (2018) Non-invasive quantification of foliar pigments: possibilities and limitations of reflectance-and absorbance-based approaches. J Photochem Photobiol B Biol 178:537–544. https://doi.org/10.1016/j.jphotobiol.2017.11.023
Gitelson AA, Keydan GP, Merzlyak MN (2006) Three-band model for noninvasive estimation of chlorophyll, carotenoids, and anthocyanin contents in higher plant leaves. Geophys Res Lett 33:L11402. https://doi.org/10.1029/2006GL026457
Häni N, Roy P, Isler V (2020) MinneApple: a benchmark dataset for apple detection and segmentation IEEE. Robot Autom Lett 5:852–858. https://doi.org/10.1109/LRA.2020.2965061
Hank TB, Berger K, Bach H, Clevers JGPW, Gitelson A, Zarco-Tejada P, Mauser W (2018) Spaceborne imaging spectroscopy for sustainable agriculture: contributions and challenges. Surv Geophys. https://doi.org/10.1007/s10712-018-9492-0
Haralick RM, Shanmugam K, Dinstein IH (1973) Textural features for image classification IEEE Transactions on systems, man, and cybernetics 610–621. https://doi.org/10.1109/TSMC.1973.4309314
Harwood D, Ojala T, Pietikäinen M, Kelman S, Davis L (1995) Texture classification by center-symmetric auto-correlation, using Kullback discrimination of distributions. Pattern Recogn Lett 16:1–10. https://doi.org/10.1016/0167-8655(94)00061-7
He K, Gkioxari G, Dollár P (2017) Girshick R Mask r-cnn. Proceedings of the IEEE international conference on computer vision, In, pp 2961–2969
Hinton GE, Salakhutdinov RR (2006) Reducing the dimensionality of data with neural networks. Science 313:504–507
Hinton O (2006) Teh, 2006 Hinton GE, Osindero S., Teh Y.-W A fast learning algorithm for deep belief nets. Neural Comput 18:1527–1554
Jin X et al (2020) High-throughput estimation of crop traits: a review of ground and aerial phenotyping platforms. IEEE Geosci Remote Sens Mag :0-0. https://doi.org/10.1109/mgrs.2020.2998816
Kior A, Sukhov V, Sukhova E (2021) Application of reflectance indices for remote sensing of plants and revealing actions of stressors. Photonics 8:582. https://doi.org/10.3390/photonics8120582
Kupidura P (2019) The comparison of different methods of texture analysis for their efficacy for land use classification in satellite imagery. Remote Sens 11:1233. https://doi.org/10.3390/rs11101233
Li H, Guo W, Lu G, Shi Y (2022a) Augmentation method for high intra-class variation data in apple detection. Sensors 22:6325. https://doi.org/10.3390/s22176325
Li T, Feng Q, Qiu Q, Xie F, Zhao C (2022b) Occluded apple fruit detection and localization with a frustum-based point-cloud-processing approach for robotic harvesting. Remote Sens 14:482. https://doi.org/10.3390/rs14030482
Li Y, Zhang H, Shen Q (2017) Spectral–spatial classification of hyperspectral imagery with 3D convolutional neural network. Remote Sens 9:67. https://doi.org/10.3390/rs9010067
Lichtenthaler HK, Babani F (2004) Light adaptation and senescence of the photosynthetic apparatus. Changes in pigment composition, chlorophyll fluorescence parameters and photosynthetic activity. In: Chlorophyll a Fluorescence. Adv Photosynth Respiration:713–736. https://doi.org/10.1007/978-1-4020-3218-9_28
Liu C, Li J, He L, Plaza A, Li S, Li B (2020) Naive Gabor networks for hyperspectral image classification. IEEE Trans Neural Netw Learn Syst 32:376–390. https://doi.org/10.1109/TNNLS.2020.2978760
Long J, Shelhamer E (2015) Darrell T Fully convolutional networks for semantic segmentation. Proceedings of the IEEE conference on computer vision and pattern recognition, In, pp 3431–3440
Maheswari P, Raja P, Apolo-Apolo OE, Pérez-Ruiz M (2021) Intelligent fruit yield estimation for orchards using deep learning based semantic segmentation techniques—a review Frontiers in. Plant Sci 12:684328. https://doi.org/10.3389/fpls.2021.684328
Makantasis K, Karantzalos K, Doulamis A, Doulamis N (2015) Deep supervised learning for hyperspectral data classification through convolutional neural networks. In: 2015 IEEE international geoscience and remote sensing symposium (IGARSS). IEEE, pp 4959–4962. https://doi.org/10.1109/IGARSS.2015.7326945
Mallat SG (1989) A theory for multiresolution signal decomposition: the wavelet representation. IEEE Trans Pattern Anal Mach Intell 11:674–693. https://doi.org/10.1109/34.192463
Mendoza F, Lu R, Ariana D, Cen H, Bailey B (2011) Integrated spectral and image analysis of hyperspectral scattering data for prediction of apple fruit firmness and soluble solids content. Postharvest Biol Technol 62:149–160. https://doi.org/10.1016/j.postharvbio.2011.05.009
Neumann M, Hallau L, Klatt B, Kersting K, Bauckhage C (2014) Erosion band features for cell phone image based plant disease classification. In: 2014 22nd International Conference on Pattern Recognition. IEEE, pp 3315–3320. https://doi.org/10.1109/ICPR.2014.571
Nguyen VD et al (2021) Noninvasive imaging technologies in plant phenotyping. Trends Plant Sci. https://doi.org/10.1016/j.tplants.2021.06.009
Oseledets IV (2011) Tensor-train decomposition. SIAM. J Sci Comput 33:2295–2317. https://doi.org/10.1137/090752286
Paoletti ME, Haut JM, Plaza J, Plaza A (2018) A new deep convolutional neural network for fast hyperspectral image classification. ISPRS J Photogramm Remote Sens 145:120–147. https://doi.org/10.1016/j.isprsjprs.2017.11.021
Ramesh S, Hebbar R, Niveditha M, Pooja R, Shashank N, Vinod P (2018) Plant disease detection using machine learning. In: 2018 International conference on design innovations for 3Cs compute communicate control (ICDI3C). IEEE, pp 41–45. https://doi.org/10.1109/ICDI3C.2018.00017
Redmon J, Divvala S, Girshick R (2016) Farhadi A You only look once: unified, real-time object detection. Proceedings of the IEEE conference on computer vision and pattern recognition, pp 779–788
Ronneberger O, Fischer P, Brox T (2015) U-net: convolutional networks for biomedical image segmentation. In: Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18. Springer, Cham. pp 234–241. https://doi.org/10.1007/978-3-319-24574-4_28
Shurygin B, Smirnov I, Chilikin A, Khort D, Kutyrev A, Zhukovskaya S, Solovchenko A (2022) Mutual augmentation of spectral sensing and machine learning for non-invasive detection of apple fruit damages. Horticulturae 8:1111. https://doi.org/10.3390/horticulturae8121111
Singh A, Ganapathysubramanian B, Singh AK, Sarkar S (2016) Machine learning for high-throughput stress phenotyping in plants trends. Plant Sci 21:110–124. https://doi.org/10.1016/j.tplants.2015.10.015
Solovchenko A, Lukyanov A, Nikolenko A, Shurygin B, Akimov M, Gitelson A (2021) Physiological foundations of spectral imaging-based monitoring of apple fruit ripening. Acta Hortic:419–428. https://doi.org/10.17660/ActaHortic.2021.1314.52
Solovchenko A, Shurygin B, Kuzin A, Solovchenko O, Krylov A (2022) Extraction of quantitative information from hyperspectral reflectance images for noninvasive plant phenotyping Russian. J Plant Physiol 69:144. https://doi.org/10.1134/S1021443722601148
Sothe C et al (2020) Comparative performance of convolutional neural network, weighted and conventional support vector machine and random forest for classifying tree species using hyperspectral and photogrammetric data. GISci Remote Sens 57:369–394. https://doi.org/10.1080/15481603.2020.1712102
Sun M, Xu L, Chen X, Ji Z, Zheng Y, Jia W (2022) Bfp net: balanced feature pyramid network for small apple detection in complex orchard environment. Plant Phenomics 2022:9892464. https://doi.org/10.34133/2022/9892464
Tang Y, Qiu J, Zhang Y, Wu D, Cao Y, Zhao K, Zhu L (2023) Optimization strategies of fruit detection to overcome the challenge of unstructured background in field orchard environment: a review. Precis Agric 24:1183–1219. https://doi.org/10.1007/s11119-023-10009-9
Van de Wouwer G, Scheunders P, Van Dyck D (1999) Statistical texture characterization from discrete wavelet representations. IEEE Trans Image Process 8:592–598. https://doi.org/10.1109/83.753747
Vetterli M, Herley C (1992) Wavelets and filter banks: theory and design. IEEE Trans Signal Process 40:2207–2232. https://doi.org/10.1109/78.157221
Wang A, Xing S, Zhao Y, Wu H, Iwahori Y (2022a) A hyperspectral image classification method based on adaptive spectral spatial kernel combined with improved vision transformer. Remote Sens 14:3705. https://doi.org/10.3390/rs14153705
Wang L, Zhang J, Liu Y, Mi J, Zhang J (2021) Multimodal medical image fusion based on Gabor representation combination of multi-CNN and fuzzy neural network IEEE. Access 9:67634–67647. https://doi.org/10.1109/ACCESS.2021.3075953
Wang M et al (2023) Tensor decompositions for hyperspectral data processing in remote sensing: a comprehensive review. IEEE Geosci Remote Sens Mag 11:26–72. https://doi.org/10.1109/MGRS.2022.3227063
Wang Y, Chen X, Cao L, Huang W, Sun F (2022b) Wang Y Multimodal token fusion for vision transformers. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, In, pp 12186–12195
Watt M, Fiorani F, Usadel B, Rascher U, Muller O, Schurr U (2020) Phenotyping: new windows into the plant for breeders. Annu Rev Plant Biol. https://doi.org/10.1146/annurev-arplant-042916-041124
Weng S, Yu S, Guo B, Tang P, Liang D (2020) Non-destructive detection of strawberry quality using multi-features of hyperspectral imaging and multivariate methods. Sensors 20:3074. https://doi.org/10.3390/s20113074
Xue Z, Tan X, Yu X, Liu B, Yu A, Zhang P (2022) Deep hierarchical vision transformer for hyperspectral and lidar data classification. IEEE Trans Image Process 31:3095–3110. https://doi.org/10.1109/TIP.2022.3162964
Ye D, Wu L, Li X, Atoba TO, Wu W, Weng H (2023) A synthetic review of various dimensions of non-destructive plant stress phenotyping. Plants (Basel):12. https://doi.org/10.3390/plants12081698
Yudina L et al (2022) Ratio of intensities of blue and red light at cultivation influences photosynthetic light reactions, respiration, growth, and reflectance indices in lettuce. Biology 11:60. https://doi.org/10.3390/biology11010060
Zavafer A, Bates H, Mancilla C, Ralph PJ (2023) Phenomics: conceptualization and importance for plant physiology. Trends Plant Sci. https://doi.org/10.1016/j.tplants.2023.03.023
Zhang C et al (2023) Feasibility assessment of tree-level flower intensity quantification from UAV RGB imagery: a triennial study in an apple orchard. ISPRS J Photogramm Remote Sens 197:256–273. https://doi.org/10.1016/j.isprsjprs.2023.02.003
Zhang J, Su R, Fu Q, Ren W, Heide F, Nie Y (2022) A survey on computational spectral reconstruction methods from RGB to hyperspectral imaging. Sci Rep 12:11905. https://doi.org/10.1038/s41598-022-16223-1
Zhang L, Zhang L, Tao D, Huang X (2011) On combining multiple features for hyperspectral remote sensing image classification. IEEE Trans Geosci Remote Sens 50:879–893
Zhao Q, Zhou G, Xie S, Zhang L, Cichocki A (2016) Tensor ring decomposition arXiv preprint arXiv:160605535
Zou Z, Chen K, Shi Z, Guo Y, Ye J (2023) Object detection in 20 years: a survey. Proc IEEE 111:257–276. https://doi.org/10.1109/JPROC.2023.3238524
Funding
Financial support of the Scientific and Educational School of Lomonosov Moscow State University “Brain, Cognitive Systems, and Artificial Intelligence” is gratefully acknowledged.
Author information
Authors and Affiliations
Contributions
All authors contributed to the study’s conception and design. The first draft of the manuscript was written by Alexei Solovchenko, and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.
Corresponding author
Ethics declarations
Ethical approval
This article does not contain any studies with animals or human participants performed by any of the authors.
Consent to participate
Not applicable.
Consent for publication
Not applicable.
Conflict of interest
The authors declare no competing interests.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Solovchenko, A., Shurygin, B., Nesterov, D.A. et al. Towards the synthesis of spectral imaging and machine learning-based approaches for non-invasive phenotyping of plants. Biophys Rev 15, 939–946 (2023). https://doi.org/10.1007/s12551-023-01125-x
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12551-023-01125-x