Abstract
The precision monitoring of film-mulched winter wheat growth facilitates field management optimization and further improves yield. Unmanned aerial vehicle (UAV) is an effective tool for crop monitoring at the field scale. However, due to the interference of background effects caused by soil and mulch, achieving accurate monitoring of crop growth in complex backgrounds for UAV remains a challenge. Additionally, the simultaneous inversion of multiple growth parameters helped us to comprehensively monitor the overall crop growth status. This study conducted field experiments including three winter wheat mulching treatments: ridge mulching, ridge–furrow full-mulching, and flat cropping full-mulching. Three machine learning algorithms (partial least squares, ridge regression, and support vector machines) and deep neural network were employed to process the vegetation indices (VIs) feature data, and the residual neural network 50 (ResNet 50) was used to process the image data. Then the two modalities (VI feature data and image data) were fused to obtain a multi-modal fusion (MMF) model. Meanwhile, a film-mulched winter wheat growth monitoring model that simultaneously predicted leaf area index (LAI), aboveground biomass (AGB), plant height (PH), and leaf chlorophyll content (LCC) was constructed by coupling multi-task learning techniques. The results showed that the image-based ResNet 50 outperformed the VI feature-based model. The MMF improved prediction accuracy for LAI, AGB, PH, and LCC with coefficients of determination of 0.73–0.92, mean absolute errors of 0.29–3.89 and relative root mean square errors of 9.48–12.99%. A multi-task MMF model with the same loss weight distribution ([1/4, 1/4, 1/4, 1/4]) achieved comparable accuracy to the single-task MMF model, improving training efficiency and providing excellent generalization to different film-mulched sample areas. The novel technique of the multi-task MMF model proposed in this study provides an accurate and comprehensive method for monitoring the growth status of film-mulched winter wheat.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data availability
The data that support the findings of this study are available from the corresponding author upon reasonable request.
References
Babaeian, E., Paheding, S., Siddique, N., Devabhaktuni, V. K., & Tuller, M. (2021). Estimation of root zone soil moisture from ground and remotely sensed soil information with multisensor data fusion and automated machine learning. Remote Sensing of Environment, 260, 112434. https://doi.org/10.1016/j.rse.2021.112434
Cai, W., Gu, X., Du, Y., Chang, T., Lu, S., Zheng, X., Bai, D., Song, H., Sun, S., & Cai, H. (2022). Effects of mulching on water saving, yield increase and emission reduction for maize in China. Agricultural Water Management, 274, 107954. https://doi.org/10.1016/j.agwat.2022.107954
Chen, P., & Wang, F. (2022). Effect of crop spectra purification on plant nitrogen concentration estimations performed using high-spatial-resolution images obtained with unmanned aerial vehicles. Field Crops Res., 288, 108708. https://doi.org/10.1016/j.fcr.2022.108708
Cheng, Z., Gu, X., Du, Y., Zhou, Z., Li, W., Zheng, X., Cai, W., & Chang, T. (2023). Spectral purification improves monitoring accuracy of the comprehensive growth evaluation index for film-mulched winter wheat. Journal of Integrative Agriculture. https://doi.org/10.1016/j.jia.2023.05.036
Cheng, Z., Gu, X., Zhou, Z., Zhang, Y., Yin, H., Li, W., Chang, T., & Du, Y. (2024). Enhancing in-season yield forecast accuracy for film-mulched wheat: A hybrid approach coupling crop model and UAV remote-sensing data by ensemble learning technique. European Journal of Agronomy, 156, 127174. https://doi.org/10.1016/j.eja.2024.127174
Daughtry, C. S. T., Walthall, C. L., Kim, M. S., de Colstoun, E. B., & McMurtrey, J. E. (2000). Estimating corn leaf chlorophyll concentration from leaf and canopy reflectance. Remote Sensing of Environment, 74, 229–239. https://doi.org/10.1016/S0034-4257(00)00113-9
Ding, J., Wu, J., Ding, D., Yang, Y., Gao, C., & Hu, W. (2021). Effects of tillage and straw mulching on the crop productivity and hydrothermal resource utilization in a winter wheat-summer maize rotation system. Agricultural Water Management, 254, 106933. https://doi.org/10.1016/j.agwat.2021.106933
Dong, T., Liu, J., Qian, B., He, L., Liu, J., Wang, R., Jing, Q., Champagne, C., McNairn, H., & Powers, J. (2020). Estimating crop biomass using leaf area index derived from Landsat 8 and Sentinel-2 data. ISPRS Journal of Photogrammetry and Remote Sensing, 168, 236–250. https://doi.org/10.1016/j.isprsjprs.2020.08.003
Elsherbiny, O., Zhou, L., He, Y., & Qiu, Z. (2022). A novel hybrid deep network for diagnosing water status in wheat crop using IoT-based multimodal data. Computers and Electronics in Agriculture, 203, 107453. https://doi.org/10.1016/j.compag.2022.107453
Fang, H., Li, Y. P., Gu, X., Yu, M., Du, Y., Chen, P., & Li, Y. N. (2021). Evapotranspiration partitioning, water use efficiency, and maize yield under different film mulching and nitrogen application in northwest China. Field Crops Res., 264, 108103. https://doi.org/10.1016/j.fcr.2021.108103
Fang, H., Liu, F., Gu, X., Chen, P., Li, Y. P., & Li, Y. N. (2022). The effect of source–sink on yield and water use of winter wheat under ridge-furrow with film mulching and nitrogen fertilization. Agricultural Water Management, 267, 107616. https://doi.org/10.1016/j.agwat.2022.107616
Fei, S., Chen, Z., Li, L., Ma, Y., & Xiao, Y. (2023). Bayesian model averaging to improve the yield prediction in wheat breeding trials. Agricultural and Forest Meteorology, 328, 109237. https://doi.org/10.1016/j.agrformet.2022.109237
Friedman, S. P. (2024). Relationships between combined and individual field crops’ biomass and planting density. Field Crops Research, 305, 109188. https://doi.org/10.1016/j.fcr.2023.109188
Gao, H., Yan, C., Liu, Q., Ding, W., Chen, B., & Li, Z. (2019). Effects of plastic mulching and plastic residue on agricultural production: A meta-analysis. Science of the Total Environment, 651, 484–492. https://doi.org/10.1016/j.scitotenv.2018.09.105
Gilabert, M. A., González-Piqueras, J., & Garcı́a-Haro, F.J., Meliá, J.,. (2002). A generalized soil-adjusted vegetation index. Remote Sensing of Environment, 82, 303–310. https://doi.org/10.1016/S0034-4257(02)00048-2
Gitelson, A. A., & Gritz †, Y., Merzlyak, M.N.,. (2003). Relationships between leaf chlorophyll content and spectral reflectance and algorithms for non-destructive chlorophyll assessment in higher plant leaves. Journal of Plant Physiology, 160, 271–282. https://doi.org/10.1078/0176-1617-00887
Gu, X., Cai, H., Chen, P., Li, Y. P., Fang, H., & Li, Y. N. (2021). Ridge-furrow film mulching improves water and nitrogen use efficiencies under reduced irrigation and nitrogen applications in wheat field. Field Crops Research, 270, 108214. https://doi.org/10.1016/j.fcr.2021.108214
Gu, X., Cheng, Z., Zhou, Z., Chang, T., Li, W., & Du, Y. (2023). Remote sensing inversion of leaf area index of mulched winter wheat based on feature downscaling and machine learning. Transactions of the Chinese Society for Agricultural Machinery (in Chinese), 54, 148–157, 167. https://doi.org/10.6041/j.issn.1000-1298.2023.06.015
Guo, Y., Chen, S., Fu, Y. H., Xiao, Y., Wu, W., Wang, H., & de Beurs, K. (2022). Comparison of multi-methods for identifying maize phenology using phenocams. Remote Sensing, 14, 244. https://doi.org/10.3390/rs14020244
Hasituya, & Chen, Z. (2017). Mapping plastic-mulched farmland with multi-temporal Landsat-8 data. Remote Sensing, 9, 557. https://doi.org/10.3390/rs9060557
He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. IEEE Conference on Computer Vision and Pattern Recognition. https://doi.org/10.1109/CVPR.2016.90
Ilniyaz, O., Du, Q., Shen, H., He, W., Feng, L., Azadi, H., Kurban, A., & Chen, X. (2023). Leaf area index estimation of pergola-trained vineyards in arid regions using classical and deep learning methods based on UAV-based RGB images. Computers and Electronics in Agriculture, 207, 107723. https://doi.org/10.1016/j.compag.2023.107723
Ji, Y., Liu, R., Xiao, Y., Cui, Y., Chen, Z., Zong, X., & Yang, T. (2023). Faba bean above-ground biomass and bean yield estimation based on consumer-grade unmanned aerial vehicle RGB images and ensemble learning. Precision Agriculture, 24, 1439–1460. https://doi.org/10.1007/s11119-023-09997-5
Jin, X., Liu, S., Baret, F., Hemerlé, M., & Comar, A. (2017). Estimates of plant density of wheat crops at emergence from very low altitude UAV imagery. Remote Sensing of Environment, 198, 105–114. https://doi.org/10.1016/j.rse.2017.06.007
Kattenborn, T., Leitloff, J., Schiefer, F., & Hinz, S. (2021). Review on convolutional neural networks in vegetation remote sensing. ISPRS Journal of Photogrammetry and Remote Sensing, 173, 24–49. https://doi.org/10.1016/j.isprsjprs.2020.12.010
Lee, H., Wang, J., & Leblon, B. (2020). Using linear regression, random forests, and support vector machine with unmanned aerial vehicle multispectral images to predict canopy nitrogen weight in corn. Remote Sens., 12, 2071. https://doi.org/10.3390/rs12132071
Li, B., Xu, X., Zhang, L., Han, J., Bian, C., Li, G., Liu, J., & Jin, L. (2020). Above-ground biomass estimation and yield prediction in potato by using UAV-based RGB and hyperspectral imaging. ISPRS Journal of Photogrammetry and Remote Sensing, 162, 161–172. https://doi.org/10.1016/j.isprsjprs.2020.02.013
Li, D., Shang, Y., He, W., & Chen, C. (2015). EXR: Greening data center network with software defined exclusive routing. IEEE Transactions on Computers, 64, 2534–2544. https://doi.org/10.1109/INFOCOM.2014.6848112
Li, D., Chen, J., Yan, Y., Zheng, H., Yao, X., Zhu, Y., Cao, W., & Cheng, T. (2022). Estimating leaf nitrogen content by coupling a nitrogen allocation model with canopy reflectance. Remote Sensing of Environment, 283, 113314. https://doi.org/10.1016/j.rse.2022.113314
Lin, T., Zhong, R., Wang, Y., Xu, J. F., Jiang, H., Xu, J. L., Ying, Y., Rodriguez, L., Ting, K. C., & Li, H. (2020). DeepCropNet: A deep spatial-temporal learning framework for county-level corn yield estimation. Environmental Research Letters, 15, 034016. https://doi.org/10.1088/1748-9326/ab66cb
Liu, Z., Wang, Y., Ata-UI-Karim, S. T., Liu, X., Tian, Y., Zhu, Y., Cao, W., & Cao, Q. (2023). Developing a nitrogen application estimation model for diverse wheat fields: A user-friendly approach for smallholder nitrogen fertilizer recommendations. European Journal of Agronomy, 151, 126984. https://doi.org/10.1016/j.eja.2023.126984
Ma, J., Liu, B., Ji, L., Zhu, Z., Wu, Y., & Jiao, W. (2023). Field-scale yield prediction of winter wheat under different irrigation regimes based on dynamic fusion of multimodal UAV imagery. International Journal of Applied Earth Observation and Geoinformation, 118, 103292. https://doi.org/10.1016/j.jag.2023.103292
Maimaitijiang, M., Sagan, V., Sidike, P., Hartling, S., Esposito, F., & Fritschi, F. B. (2020). Soybean yield prediction from UAV using multimodal data fusion and deep learning. Remote Sensing of Environment, 237, 111599. https://doi.org/10.1016/j.rse.2019.111599
Narmilan, A., Gonzalez, F., Salgadoe, A. S. A., Kumarasiri, U. W. L. M., Weerasinghe, H. A. S., & Kulasekara, B. R. (2022). Predicting canopy chlorophyll content in sugarcane crops using machine learning algorithms and spectral vegetation indices derived from UAV multispectral imagery. Remote Sensing, 14, 1140. https://doi.org/10.3390/rs14051140
Qi, H., Wu, Z., Zhang, L., Li, J., Zhou, J., Jun, Z., & Zhu, B. (2021). Monitoring of peanut leaves chlorophyll content based on drone-based multispectral image feature extraction. Computers and Electronics in Agriculture, 187, 106292. https://doi.org/10.1016/j.compag.2021.106292
Qiao, L., Gao, D., Zhao, R., Tang, W., An, L., Li, M., & Sun, H. (2022a). Improving estimation of LAI dynamic by fusion of morphological and vegetation indices based on UAV imagery. Computers and Electronics in Agriculture, 192, 106603. https://doi.org/10.1016/j.compag.2021.106603
Qiao, L., Zhao, R., Tang, W., An, L., Sun, H., Li, M., Wang, N., Liu, Y., & Liu, G. (2022b). Estimating maize LAI by exploring deep features of vegetation index map from UAV multispectral images. Field Crops Research, 289, 108739. https://doi.org/10.1016/j.fcr.2022.108739
Reyniers, M., Walvoort, D. J. J., & De Baardemaaker, J. (2006). A linear model to predict with a multi-spectral radiometer the amount of nitrogen in winter wheat. International Journal of Remote Sensing, 27, 4159–4179. https://doi.org/10.1080/01431160600791650
Rondeaux, G., Steven, M., & Baret, F. (1996). Optimization of soil-adjusted vegetation indices. Remote Sensing of Environment, 55, 95–107. https://doi.org/10.1016/0034-4257(95)00186-7
Ruan, G., Schmidhalter, U., Yuan, F., Cammarano, D., Liu, X., Tian, Y., Zhu, Y., Cao, W., & Cao, Q. (2023). Exploring the transferability of wheat nitrogen status estimation with multisource data and Evolutionary Algorithm-Deep Learning (EA-DL) framework. European Journal of Agronomy, 143, 126727. https://doi.org/10.1016/j.eja.2022.126727
Schneider, P., Roberts, D. A., & Kyriakidis, P. C. (2008). A VARI-based relative greenness from MODIS data for computing the Fire Potential Index. Remote Sensing of Environment, 112, 1151–1167. https://doi.org/10.1016/j.rse.2007.07.010
Shao, G., Han, W., Zhang, H., Liu, S., Wang, Y., Zhang, L., & Cui, X. (2021). Mapping maize crop coefficient Kc using random forest algorithm based on leaf area index and UAV-based multispectral vegetation indices. Agricultural Water Management, 252, 106906. https://doi.org/10.1016/j.agwat.2021.106906
Sun, Z., Li, Q., Jin, S., Song, Y., Xu, S., Wang, X., Cai, J., Zhou, Q., Ge, Y., Zhang, R., Zang, J., & Jiang, D. (2022). Simultaneous prediction of wheat yield and grain protein content using multitask deep learning from time-series proximal sensing. Plant Phenomics, 2022, 9757948. https://doi.org/10.34133/2022/9757948
Tanabe, R., Matsui, T., & Tanaka, T. S. T. (2023). Winter wheat yield prediction using convolutional neural networks and UAV-based multispectral imagery. Field Crops Research, 291, 108786. https://doi.org/10.1016/j.fcr.2022.108786
Wan, L., Cen, H., Zhu, J., Zhang, J., Zhu, Y., Sun, D., Du, X., Zhai, L., Weng, H., Li, Y., Li, X., Bao, Y., Shou, J., & He, Y. (2020). Grain yield prediction of rice using multi-temporal UAV-based RGB and multispectral images and model transfer—a case study of small farmlands in the South of China. Agricultural and Forest Meteorology, 291, 108096. https://doi.org/10.1016/j.agrformet.2020.108096
Wang, W., Zheng, H., Wu, Y., Yao, X., Zhu, Y., Cao, W., & Cheng, T. (2022). An assessment of background removal approaches for improved estimation of rice leaf nitrogen concentration with unmanned aerial vehicle multispectral imagery at various observation times. Field Crops Research, 283, 108543. https://doi.org/10.1016/j.fcr.2022.108543
Wittstruck, L., Jarmer, T., Trautz, D., & Waske, B. (2022). Estimating LAI from winter wheat using UAV data and CNNs. IEEE Geoscience and Remote Sensing Letters. https://doi.org/10.1109/LGRS.2022.3141497
Xiao, Z., Liang, S., Wang, J., Xiang, Y., Zhao, X., & Song, J. (2016). Long-time-series global land surface satellite leaf area index product derived from MODIS and AVHRR surface reflectance. IEEE Transactions on Geoscience and Remote Sensing, 54(9), 5301–5318. https://doi.org/10.1109/TGRS.2016.2560522
Xu, X., Nie, C., Jin, X., Li, Z., Zhu, H., Xu, H., Wang, J., Zhao, Y., & Feng, H. (2021). A comprehensive yield evaluation indicator based on an improved fuzzy comprehensive evaluation method and hyperspectral data. Field Crops Research, 270, 108204. https://doi.org/10.1016/j.fcr.2021.108204
Xue, J., & Su, B. (2017). Significant remote sensing vegetation indices: A review of developments and applications. J Sens., 2017, 1353691. https://doi.org/10.1155/2017/1353691
Yamaguchi, T., Tanaka, Y., Imachi, Y., Yamashita, M., & Katsura, K. (2020). Feasibility of combining deep learning and RGB images obtained by unmanned aerial vehicle for leaf area index estimation in rice. Remote Sensing, 13, 84. https://doi.org/10.3390/rs13010084
Yang, Q., Shi, L., Han, J., Zha, Y., & Zhu, P. (2019). Deep convolutional neural networks for rice grain yield estimation at the ripening stage using UAV-based remotely sensed images. Field Crops Research, 235, 142–153. https://doi.org/10.1016/j.fcr.2019.02.022
Yu, D., Zha, Y., Sun, Z., Li, J., Jin, X., Zhu, W., Bian, J., Ma, L., Zeng, Y., & Su, Z. (2022). Deep convolutional neural networks for estimating maize above-ground biomass using multi-source UAV images: a comparison with traditional machine learning algorithms. Precision Agriculture, 24, 92–113. https://doi.org/10.1007/s11119-022-09932-0
Zhang, Y., Han, W., Zhang, H., Niu, X., & Shao, G. (2023). Evaluating soil moisture content under maize coverage using UAV multimodal data by machine learning algorithms. Journal of Hydrology, 617, 129086. https://doi.org/10.1016/j.jhydrol.2023.129086
Zhang, Y., Hui, J., Qin, Q., Sun, Y., Zhang, T., Sun, H., & Li, M. (2021). Transfer-learning-based approach for leaf chlorophyll content estimation of winter wheat from hyperspectral data. Remote Sensing of Environment, 267, 112724. https://doi.org/10.1016/j.rse.2021.112724
Zhao, X., Gu, X., Yang, Z., Li, Y., Zhang, L., & Zhou, J. (2022). Effects of soil preparation and mulching practices together with different urea applications on the water and nitrogen use of winter wheat in semi-humid and drought-prone areas. Agricultural Water Management, 263, 107484. https://doi.org/10.1016/j.agwat.2022.107484
Zhou, J., Li, J., Wang, C., Wu, H., Zhao, C., & Teng, G. (2021a). Crop disease identification and interpretation method based on multimodal deep learning. Computers and Electronics in Agriculture, 189, 106408. https://doi.org/10.1016/j.compag.2021.106408
Zhou, J., Zhou, J. F., Ye, H., Ali, M. L., Chen, P., & Nguyen, H. T. (2021b). Yield estimation of soybean breeding lines under drought stress using unmanned aerial vehicle-based imagery and convolutional neural network. Biosystems Engineering, 204, 90–103. https://doi.org/10.1016/j.biosystemseng.2021.01.017
Zhu, W., Rezaei, E. E., Nouri, H., Sun, Z., Li, J., Yu, D., & Siebert, S. (2022). UAV-based indicators of crop growth are robust for distinct water and nutrient management but vary between crop development phases. Field Crops Research, 284, 108582. https://doi.org/10.1016/j.fcr.2022.108582
Acknowledgements
This study was funded by the National Key R&D Program of China (2021YFD1900700), Key R&D Program of Shaanxi Province, China (2023-YBNY-040 and 2022NY-114), National Natural Science Foundation of China (51909221), China Postdoctoral Science Foundation (2020T130541 and 2019M650277).
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Below is the link to the electronic supplementary material.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Cheng, Z., Gu, X., Du, Y. et al. Multi-modal fusion and multi-task deep learning for monitoring the growth of film-mulched winter wheat. Precision Agric (2024). https://doi.org/10.1007/s11119-024-10147-8
Accepted:
Published:
DOI: https://doi.org/10.1007/s11119-024-10147-8