Skip to main content
Book cover

Doctoral Symposium on Information and Communication Technologies

DSICT 2022: Doctoral Symposium on Information and Communication Technologies pp 106–117Cite as

Multimodal Deep Learning for Crop Yield Prediction

  • 144 Accesses

Part of the Communications in Computer and Information Science book series (CCIS,volume 1647)

Abstract

Precision agriculture is a vital practice for improving the production of crops. The present work is aimed to develop a deep learning multimodal model that can predict the crop yield in Ecuadorian corn farms. The model takes multispectral images and field sensor data (humidity, temperature, or soil status) to obtain the yield of a crop. The use of multimodal data is aimed to extract hidden patterns in the status of crops and in this way obtain better results than the use of vegetation indices or other state-of-the-art methods. For the experiments, we utilized multi-spectral satellite images obtained from the google earth engine platform and monthly precipitation and temperature data of the 24 Ecuadorian provinces collected from the Ecuadorian Ministry of agriculture and livestock; likewise, we obtained the area of corn plantation in each province and their corn production for the years 2016 to 2020. Results indicate that the use of multimodal deep learning models (pre-trained CNN for images and LSTM for time series sensor data) gives better prediction accuracy than monomodal prediction models.

Keywords

  • Precision agriculture
  • Remote sensing
  • Convolutional neural networks
  • Recurrent neural networks
  • Multimodal deep learning

This is a preview of subscription content, access via your institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-031-18347-8_9
  • Chapter length: 12 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   64.99
Price excludes VAT (USA)
  • ISBN: 978-3-031-18347-8
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   84.99
Price excludes VAT (USA)
Learn about institutional subscriptions
Fig. 1.
Fig. 2.
Fig. 3.
Fig. 4.

References

  1. Ramanath, A., Muthusrinivasan, S., Xie, Y., Shekhar, S., Ramachandra, B.: NDVI versus CNN features in deep learning for land cover classification of aerial images. In: IGARSS 2019–2019 IEEE International Geoscience and Remote Sensing Symposium, pp. 6483–6486. IEEE (2019)

    Google Scholar 

  2. Tran, T., Choi, J., Le, T., Kim, J.: A comparative study of deep CNN in forecasting and classifying the macronutrient deficiencies on development of tomato plant. Appl. Sci. 9(8), 1601 (2019)

    CrossRef  Google Scholar 

  3. Chlingaryan, A., Sukkarieh, S., Whelan, B.: Machine learning approaches for crop yield prediction and nitrogen status estimation in precision agriculture: a review. Comput. Electron. Agric. 151, 61–69 (2018)

    CrossRef  Google Scholar 

  4. Wiegand, C., Richardson, A., Escobar, D., Gerbermann, A.: Vegetation indices in crop assessments. Remote Sens. Environ. 35(2–3), 105–119 (1991)

    CrossRef  Google Scholar 

  5. Basso, B., Cammarano, D., Carfagna, E.: Review of crop yield forecasting methods and early warning systems. In: Proceedings of the First Meeting of the Scientific Advisory Committee of the Global Strategy to Improve Agricultural and Rural Statistics, FAO Headquarters, Rome, Italy, pp. 18–19 (2013)

    Google Scholar 

  6. Mahdavinejad, M., Rezvan, M., Barekatain, M., Adibi, P., Barnaghi, P., Sheth, A.: Machine learning for Internet of Things data analysis: a survey. Digit. Commun. Netw. 4(3), 161–175 (2018)

    CrossRef  Google Scholar 

  7. Gondchawar, N., Kawitkar, R.: IoT based smart agriculture. Int. J. Adv. Res. Comput. Commun. Eng. 5(6), 838–842 (2016)

    Google Scholar 

  8. Muangprathub, J., Boonnam, N., Kajornkasirat, S., Lekbangpong, N., Wanichsombat, A., Nillaor, P.: IoT and agriculture data analysis for smart farm. Comput. Electron. Agric. 156, 467–474 (2019)

    CrossRef  Google Scholar 

  9. Kim, T., Ramos, C., Mohammed, S.: Smart city and IoT (2017)

    Google Scholar 

  10. Samuel, S.: A review of connectivity challenges in IoT-smart home. In: 2016 3rd MEC International Conference on Big Data and Smart City (ICBDSC), pp. 1–4. IEEE (2016)

    Google Scholar 

  11. Kim, Y., Park, Y., Choi, J.: A study on the adoption of IoT smart home service: using value-based adoption model. Total Qual. Manag. Bus. Excell. 28(9–10), 1149–1165 (2017)

    CrossRef  Google Scholar 

  12. Ukil, A., Bandyoapdhyay, S., Puri, C., Pal, A.: IoT healthcare analytics: the importance of anomaly detection. In: 2016 IEEE 30th International Conference on Advanced Information Networking and Applications (AINA), pp. 994–997. IEEE (2016)

    Google Scholar 

  13. Tyagi, S., Agarwal, A., Maheshwari, P.: A conceptual framework for IoT-based healthcare system using cloud computing. In: 2016 6th International Conference-Cloud System and Big Data Engineering (Confluence), pp. 503–507. IEEE (2016)

    Google Scholar 

  14. Rghioui, A., Sendra, S., Lloret, J., Oumnad, A.: Internet of Things for measuring human activities in ambient assisted living and e-health. Netw. Protoc. Algorithms 8(3), 15–28 (2016)

    CrossRef  Google Scholar 

  15. Shi, C., Liu, J., Liu, H., Chen, Y.: Smart user authentication through actuation of daily activities leveraging WiFi-enabled IoT. In: Proceedings of the 18th ACM International Symposium on Mobile Ad Hoc Networking and Computing, pp. 1–10 (2017)

    Google Scholar 

  16. Al-Douri, Y.K., Hamodi, H., Lundberg, J.: Time series forecasting using a two-level multi-objective genetic algorithm: a case study of maintenance cost data for tunnel fans. Algorithms 11(8), 123 (2018)

    CrossRef  MathSciNet  MATH  Google Scholar 

  17. Baptista, M., Sankararaman, S., de Medeiros, I., Nascimento, C., Jr., Prendinger, H., Henriques, E.: Forecasting fault events for predictive maintenance using data-driven techniques and ARMA modeling. Comput. Ind. Eng. 115, 41–53 (2018)

    CrossRef  Google Scholar 

  18. Kamir, E., Waldner, F., Hochman, Z.: Estimating wheat yields in Australia using climate records, satellite image time series and machine learning methods. ISPRS J. Photogramm. Remote. Sens. 160, 124–135 (2020)

    CrossRef  Google Scholar 

  19. Adeniyi, O.D., Szabo, A., Tamás, J., Nagy, A.: Wheat Yield Forecasting Based on Landsat NDVI and SAVI Time Series (2020)

    Google Scholar 

  20. Kadri, F., Harrou, F., Chaabane, S., Tahon, C.: Time series modelling and forecasting of emergency department overcrowding. J. Med. Syst. 38(9), 1–20 (2014). https://doi.org/10.1007/s10916-014-0107-0

    CrossRef  Google Scholar 

  21. Demir, E., Dincer, S.: Place and solution proposals of data mining in production planning and control processes: a business application. Press Academia Procedia 11(1), 189–193 (2020)

    Google Scholar 

  22. Srivastava, N., Salakhutdinov, R.: Multimodal learning with deep Boltzmann machines. In: Advances in Neural Information Processing Systems, vol. 25 (2012)

    Google Scholar 

  23. Ramachandram, D., Taylor, G.: Deep multimodal learning: a survey on recent advances and trends. IEEE Sig. Process. Mag. 34(6), 96–108 (2017)

    CrossRef  Google Scholar 

  24. Maimaitijiang, M., Sagan, V., Sidike, P., Hartling, S., Esposito, F., Fritschi, F.: Soybean yield prediction from UAV using multimodal data fusion and deep learning. Remote Sens. Environ. 237, 111599 (2020)

    CrossRef  Google Scholar 

  25. Yalcin, H.: Plant phenology recognition using deep learning: Deep-Pheno. In: 2017 6th International Conference on Agro-Geoinformatics, pp. 1–5. IEEE (2017)

    Google Scholar 

  26. Zheng, Y.Y., Kong, J.L., Jin, X.B., Wang, X.Y., Su, T.L., Zuo, M.: CropDeep: the crop vision dataset for deep-learning-based classification and detection in precision agriculture. Sensors 19(5), 1058 (2019)

    CrossRef  Google Scholar 

  27. Nilsback, M., Zisserman, A.: A visual vocabulary for flower classification. In: Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2006), New York, NY, USA, pp. 1447–1454 (2006)

    Google Scholar 

  28. Kumar, N., et al.: Leafsnap: a computer vision system for automatic plant species identification. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) Computer Vision, vol. 7573, pp. 502–516. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33709-3_36

    CrossRef  Google Scholar 

  29. Wegner, J., Branson, S., Hall, D., Schindler, K., Perona, P.: Cataloging public objects using aerial and street-level images-urban trees. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas Valley, NV, USA, pp. 6014–6023 (2016)

    Google Scholar 

  30. Kamilaris, A., Prenafeta-Boldú, F.: Deep learning in agriculture: a survey. Comput. Electron. Agric. 147, 70–90 (2018)

    CrossRef  Google Scholar 

  31. Bender, A., Whelan, B., Sukkarieh, S.: Ladybird Cobbitty 2017 Brassica dataset (2019)

    Google Scholar 

  32. Gandhi, A., Sharma, A., Biswas, A., Deshmukh, O.: GeThR-Net: a generalized temporally hybrid recurrent neural network for multimodal information fusion. In: Hua, G., Jégou, H. (eds.) Computer Vision, vol. 9914, pp. 883–899. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-48881-3_58

    CrossRef  Google Scholar 

  33. Gao, J., Li, P., Chen, Z., Zhang, J.: A survey on deep learning for multimodal data fusion. Neural Comput. 32(5), 829–864 (2020)

    CrossRef  MathSciNet  MATH  Google Scholar 

  34. Zhao, X., et al.: Use of unmanned aerial vehicle imagery and deep learning UNet to extract rice lodging. Sensors 19(18), 3859 (2019)

    CrossRef  Google Scholar 

  35. Chen, W., Wang, W., Liu, L., Lew, M.: New ideas and trends in deep multimodal content understanding: a review. arXiv preprint https://arxiv.org/abs/2010.08189 (2020)

  36. Iniap. http://www.iniap.gob.ec/pruebav3/wp-content/uploads/2018/03/281-iniap-OK-baja.pdf

  37. Sistema de Información Pública Agropecuaria. http://sipa.agricultura.gob.ec/index.php/maiz

  38. Google Earth Engine data catalog, Sentinel-2 MSI. https://developers.google.com/earth-engine/datasets/catalog/COPERNICUS_S2

Download references

Author information

Authors and Affiliations

  1. Escuela Superior Politécnica del Litoral, ESPOL, Facultad de Ingeniería en Electricidad y Computación, CIDIS, Gustavo Galindo Km. 30.5 Vía Perimetral, Guayaquil, Ecuador

    Luis-Roberto Jácome-Galarza

Authors
  1. Luis-Roberto Jácome-Galarza
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Luis-Roberto Jácome-Galarza .

Editor information

Editors and Affiliations

  1. CEDIA, Cuenca, Ecuador

    Karina Abad

  2. CEDIA, Cuenca, Ecuador

    Santiago Berrezueta

Rights and permissions

Reprints and Permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Verify currency and authenticity via CrossMark

Cite this paper

Jácome-Galarza, LR. (2022). Multimodal Deep Learning for Crop Yield Prediction. In: Abad, K., Berrezueta, S. (eds) Doctoral Symposium on Information and Communication Technologies. DSICT 2022. Communications in Computer and Information Science, vol 1647. Springer, Cham. https://doi.org/10.1007/978-3-031-18347-8_9

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-18347-8_9

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-18346-1

  • Online ISBN: 978-3-031-18347-8

  • eBook Packages: Computer ScienceComputer Science (R0)