Abstract
Smallholder farmers play an important role in the global food supply. As smartphones become increasingly pervasive, they enable smallholder farmers to collect images at very low cost. In this study, an efficient deep convolutional neural network (DCNN) architecture was proposed to detect development stages (DVS) of paddy rice using photographs taken by a handheld camera. The DCNN model was trained with different strategies and compared against the traditional time series Green chromatic coordinate (time-series Gcc) method and the manually extracted feature-combining support vector machine (MF-SVM) method. Furthermore, images taken at different view angles, model training strategies, and interpretations of predictions of the DCNN models were investigated. Optimal results were obtained by the DCNN model trained with the proposed two-step fine-tuning strategy, with a high overall accuracy of 0.913 and low mean absolute error of 0.090. The results indicated that images taken at large view angles contained more valuable information and the performance of the model can be further improved by using images taken at multiple angles. The two-step fine-tuning strategy greatly improved the model robustness against the randomness of view angle. The interpretation results demonstrated that it is possible to extract phenology-related features from images. This study provides a phenology detection approach to utilize handheld camera images in real time and some important insights into the use of deep learning in real world scenarios.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Azizpour, H., Razavian, A. S., Sullivan, J., Maki, A., & Carlsson, S. (2015). From generic to specific deep representations for visual recognition. In IEEE Conference on computer vision and pattern recognition workshops, October 2015 (pp. 36–45). https://doi.org/10.1109/CVPRW.2015.7301270.
Bai, X., Cao, Z., Zhao, L., Zhang, J., Lv, C., Li, C., & Xie, J. (2018). Rice heading stage automatic observation by multi-classifier cascade based rice spike detection method. Agricultural and Forest Meteorology, 259, 21360–270. https://doi.org/10.1016/j.agrformet.2018.05.001
Chen, S. W., Shivakumar, S. S., Dcunha, S., Das, J., Okon, E., Qu, C., et al. (2017). Counting apples and oranges with deep learning: A data-driven approach. IEEE Robotics and Automation Letters, 2, 781–788. https://doi.org/10.1109/LRA.2017.2651944
Cui, Z., Zhang, H., Chen, X., Zhang, C., Ma, W., Huang, C., et al. (2018). Pursuing sustainable productivity with millions of smallholder farmers. Nature, 555, 363–366. https://doi.org/10.1038/nature25785
Dyson, J., Mancini, A., Frontoni, E., & Zingaretti, P. (2019). Deep learning for soil and crop segmentation from remotely sensed data. Remote Sensing, 11, 1859. https://doi.org/10.3390/rs11161859
Ferentinos, K. P. (2018). Deep learning models for plant disease detection and diagnosis. Computers and Electronics in Agriculture, 145, 311–318. https://doi.org/10.1016/j.compag.2018.01.009
Gebbers, R., & Adamchuk, V. I. (2010). Precision agriculture and food security. Science. https://doi.org/10.1126/science.1183899
Guo, W., Fukatsu, T., & Ninomiya, S. (2015). Automated characterization of flowering dynamics in rice using field-acquired time-series RGB images. Plant Methods, 11, 7. https://doi.org/10.1186/s13007-015-0047-9
Hope, V. M. (2012). Deep learning of representations for unsupervised and transfer learning. In Proceedings of ICML-2011 workshop on unsupervised and transfer learning (pp. 13–36). https://doi.org/10.1007/978-3-642-36657-4_1.
Huete, A., Miura, T., Yoshioka, H., Ratana, P., & Broich, M. (2013). Indices of vegetation activity. In J. M. Hanes (Ed.), Biophysical applications of satellite remote sensing (pp. 1–41). Berlin: Springer. https://doi.org/10.1007/978-3-642-25047-7_1.
Hufkens, K., Melaas, E. K., Foster, T., Robles, M., Mann, M. L., Kramer, B., & Ceballos, F. (2018). Monitoring crop phenology using a smartphone based near-surface remote sensing approach. Agricultural and Forest Meteorology, 265, 327–337. https://doi.org/10.1016/j.agrformet.2018.11.002
Ichardson, A. N. D. R., Raswell, B. O. H. B., Ollinger, D. A. Y. H., & Enkins, J. U. P. J. (2009). Near-surface remote sensing of spatial and temporal variation. Ecological Applications, 19, 1417–1428.
Jamieson, P. D., Brooking, I. R., Semenov, M. A., McMaster, G. S., White, J. W., & Porter, J. R. (2007). Reconciling alternative models of phenological development in winter wheat. Field Crops Research , 103(1), 36–41. https://doi.org/10.1016/j.fcr.2007.04.009
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., & Darrell, T. (2014). Caffe: Convolutional architecture for fast feature embedding. In ACM conference on multimedia (pp. 675-678). https://doi.org/10.1145/2647868.2654889.
Kamilaris, A., & Prenafeta-Boldú, F. X. (2018). Deep learning in agriculture: A survey. Computers and Electronics in Agriculture, 147, 70–90. https://doi.org/10.1016/j.compag.2018.02.016
Klosterman, S., Melaas, E., Wang, J., Martinez, A., Frederick, S., O’Keefe, J., et al. (2018). Fine-scale perspectives on landscape phenology from unmanned aerial vehicle (UAV) photography. Agricultural and Forest Meteorology, 248, 397–407. https://doi.org/10.1016/j.agrformet.2017.10.015
Koirala, A., Walsh, K. B., Wang, Z., & McCarthy, C. (2019). Deep learning for real-time fruit detection and orchard fruit load estimation: Benchmarking of ‘MangoYOLO.’ Precision Agriculture, 20, 1107–1135. https://doi.org/10.1007/s11119-019-09642-0.
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). ImageNet classification with deep convolutional neural networks. In Handbook of approximation algorithms and metaheuristics (pp. 60-1–60-16). London: Chapman & Hall. https://doi.org/10.1201/9781420010749.
Liu, X., Chen, S. W., Aditya, S., Sivakumar, N., Dcunha, S., Qu, C., Taylor, C. J., Das, J., & Kumar, V. (2018). Robust fruit counting: Combining deep learning, tracking, and structure from motion. In IEEE international conference on intelligent robots and systems (pp. 1045–1052). https://doi.org/10.1109/IROS.2018.8594239.
Liu, X., Li, S., Kan, M., Zhang, J., Wu, S., Liu, W., Han, H., Shan, S., & Chen, X. (2015). AgeNet: Deeply learned regressor and classifier for robust apparent age estimation. In Proceedings of IEEE international conference on computer vision workshops (pp. 16–24). https://doi.org/10.1109/ICCVW.2015.42.
Lowder, S. K., Skoet, J., & Raney, T. (2016). The number, size, and distribution of farms, smallholder farms, and family farms worldwide. World Development, 87, 16–29. https://doi.org/10.1016/j.worlddev.2015.10.041
Ma, J., Li, Y., Chen, Y., Du, K., Zheng, F., Zhang, L., & Sun, Z. (2019). Estimating above ground biomass of winter wheat at early growth stages using digital images and deep convolutional neural network. European Journal of Agronomy, 103, 117–129. https://doi.org/10.1016/j.eja.2018.12.004.
Madec, S., Jin, X., Lu, H., De Solan, B., Liu, S., Duyme, F., et al. (2019). Ear density estimation from high resolution RGB imagery using deep learning technique. Agricultural and Forest Meteorology, 264, 225–234. https://doi.org/10.1016/j.agrformet.2018.10.013
Milioto, A., Lottes, P., & Stachniss, C. (2018). Real-Time semantic segmentation of crop and weed for precision agriculture robots leveraging background knowledge in CNNs. In Proceedings of IEEE international conference on robotics and automation (pp. 2229–2235). https://doi.org/10.1109/ICRA.2018.8460962.
Park, J. Y., Muller-Landau, H. C., Lichstein, J. W., Rifai, S. W., Dandois, J. P., & Bohlman, S. A. (2019). Quantifying leaf phenology of individual trees and species in a tropical forest using unmanned aerial vehicle (UAV) images. Remote Sensing, 11, 1534. https://doi.org/10.3390/rs11131534
Patrício, D. I., & Rieder, R. (2018). Computer vision and artificial intelligence in precision agriculture for grain crops: A systematic review. Computers and Electronics in Agriculture, 153, 69–81. https://doi.org/10.1016/j.compag.2018.08.001
Perez, L., & Wang, J. (2017). The effectiveness of data augmentation in image classification using deep learning. arXiv preprint. arXiv:1712.04621.
Petach, A. R., Toomey, M., Aubrecht, D. M., & Richardson, A. D. (2014). Monitoring vegetation phenology using an infrared-enabled security camera. Agricultural and Forest Meteorology, 195–196, 143–151. https://doi.org/10.1016/j.agrformet.2014.05.008
Putra, B. T. W., & Soni, P. (2019). Improving nitrogen assessment with an RGB camera across uncertain natural light from above-canopy measurements. Precision Agriculture, 13(3), 285–301. https://doi.org/10.1007/s11119-019-09656-8
Ricciardi, V., Ramankutty, N., Mehrabi, Z., & Jarvis, L. (2018). How much of the world’s food do smallholders produce? Global Food Security, 17, 64–72. https://doi.org/10.1016/j.gfs.2018.05.002
Sa, I., Popović, M., Khanna, R., Chen, Z., Lottes, P., Liebisch, F., et al. (2018). WeedMap: A large-scale semantic weed mapping framework using aerial multispectral imaging and deep neural network for precision farming. Remote Sensing, 10(9), 1423. https://doi.org/10.3390/rs10091423
Sakamoto, T., Gitelson, A. A., Wardlow, B. D., Arkebauer, T. J., Verma, S. B., Suyker, A. E., & Shibayama, M. (2012). Application of day and night digital photographs for estimating maize biophysical characteristics. Precision Agriculture, 13(3), 285–301. https://doi.org/10.1007/s11119-011-9246-1
Schwartz, M. D. (Ed.). (2013). Phenology: An integrative environmental science (pp. 548–550). Dordrecht: Kluwer.
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017). Grad-CAM: Visual Explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, October 2017 (pp. 618–626). https://doi.org/10.1109/ICCV.2017.74.
Sladojevic, S., Arsenovic, M., Anderla, A., Culibrk, D., & Stefanovic, D. (2016). Deep neural networks based recognition of plant diseases by leaf image classification. Computational Intelligence and Neuroscience, 2016, Article 3289801. https://doi.org/10.1155/2016/3289801.
Smith, A. R. (1978). Color gamut transform pairs. ACM SIGGRAPH Computer Graphics, 12, 12–19. https://doi.org/10.1145/965139.807361
Sonnentag, O., Hufkens, K., Teshera-Sterne, C., Young, A. M., Friedl, M., Braswell, B. H., et al. (2012). Digital repeat photography for phenological research in forest ecosystems. Agricultural and Forest Meteorology, 152, 159–177. https://doi.org/10.1016/j.agrformet.2011.09.009
Lancashire, P. D., Bleiholder, H., Boom, T. V. D., Langelüddeke, P., Stauss, R., WEBER, E., & Witzenberger, A. (1991). A uniform decimal code for growth stages of crops and weeds. Annals of Applied Biology, 119(3), 561–601. https://doi.org/10.1111/j.1744-7348.1991.tb04895.x
Stein, M., Bargoti, S., & Underwood, J. (2016). Image based mango fruit detection, localisation and yield estimation using multiple view geometry. Sensors (Switzerland), 16, 1915. https://doi.org/10.3390/s16111915
Sunoj, S., Igathinathane, C., & Hendrickson, J. (2016). Monitoring plant phenology using phenocam: A review. In ASABE annual international meeting (pp. 1–9). https://doi.org/10.13031/aim.20162461829.
Sutskever, I., Martens, J., Dahl, G., & Hinton, G. (2013). On the importance of initialization and momentum in deep learning. In Proceedings of the 30th international conference on machine learning (PMLR) (Vol. 28(3), pp. 1139–1147). https://doi.org/10.1017/CBO9781316423936.
Tajbakhsh, N., Shin, J. Y., Gurudu, S. R., Hurst, R. T., Kendall, C. B., Gotway, M. B., & Liang, J. (2016). Convolutional neural networks for medical image analysis: Full training or fine tuning? IEEE Transactions on Medical Imaging, 35, 1299–1312. https://doi.org/10.1109/TMI.2016.2535302
White, M. A., de Beurs, K. M., Didan, K., Inouye, D. W., Richardson, A. D., Jensen, O. P., et al. (2009). Intercomparison, interpretation, and assessment of spring phenology in North America estimated from remote sensing for 1982–2006. Global Change in Biology, 15, 2335–2359. https://doi.org/10.1111/j.1365-2486.2009.01910.x
Yalcin, H. (2017). Plant phenology recognition using deep learning: Deep-pheno. In 2017 The sixth international conference on agro-geoinformatics (pp. 1–5). https://doi.org/10.1109/Agro-Geoinformatics.2017.8046996.
Yang, Q., Shi, L., Han, J., Zha, Y., & Zhu, P. (2019). Deep convolutional neural networks for rice grain yield estimation at the ripening stage using UAV-based remotely sensed images. Field Crops Research, 235, 142–153. https://doi.org/10.1016/j.fcr.2019.02.022
Yoshida, S. (1981). Fundamentals of rice crop science. In Growth and development of the rice plant. Los Baños: International Rice Research Institute.
Yu, Z., Cao, Z., Wu, X., Bai, X., Qin, Y., Zhuo, W., et al. (2013). Automatic image-based detection technology for two critical growth stages of maize: Emergence and three-leaf stage. Agricultural and Forest Meteorology, 174–175, 65–84. https://doi.org/10.1016/j.agrformet.2013.02.011
Zhang, N., Wang, M., & Wang, N. (2002). Precision agriculture—a worldwide overview. Computers and Electronics in Agriculture, 36, 113–132. https://doi.org/10.1016/S0168-1699(02)00096-0.
Zhang, X., Friedl, M. A., & Schaaf, C. B. (2006). Global vegetation phenology from Moderate Resolution Imaging Spectroradiometer (MODIS): Evaluation of global patterns and comparison with in situ measurements. Journal of Geophysical Research: Biogeosciences, 111, 1–14. https://doi.org/10.1029/2006JG000217
Zhang, X., Jayavelu, S., Liu, L., Friedl, M. A., Henebry, G. M., Liu, Y., et al. (2018). Evaluation of land surface phenology from VIIRS data using time series of PhenoCam imagery. Agricultural and Forest Meteorology, 256–257, 137–149. https://doi.org/10.1016/j.agrformet.2018.03.003
Zheng, H., Cheng, T., Yao, X., Deng, X., Tian, Y., Cao, W., & Zhu, Y. (2016). Detection of rice phenology through time series analysis of ground-based spectral index data. Field Crops Research, 198, 131–139. https://doi.org/10.1016/j.fcr.2016.08.027
Zhu, Y., Cao, Z., Lu, H., Li, Y., & Xiao, Y. (2016). In-field automatic observation of wheat heading stage using computer vision. Biosystems Engineering, 143, 28–41. https://doi.org/10.1016/j.biosystemseng.2015.12.015
Acknowledgements
This study was supported by the National Natural Science Foundation of China Grant 51861125202 and 51629901, and the Key Research and Development Program in Guangxi Grant AB16380257.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Han, J., Shi, L., Yang, Q. et al. Real-time detection of rice phenology through convolutional neural network using handheld camera images. Precision Agric 22, 154–178 (2021). https://doi.org/10.1007/s11119-020-09734-2
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11119-020-09734-2