Skip to main content

Advertisement

SpringerLink
Log in

Early/late fusion structures with optimized feature selection for weed detection using visible and thermal images of paddy fields

  • Published:
Precision Agriculture Aims and scope Submit manuscript

Abstract

In order to reduce the overuse of herbicides, an automatic spraying system can be utilized, assisted with machine vision-based techniques to accurately target weeds. In this paper, a fusion-based structure has been proposed for weed detection in visible and thermal images of paddy fields. Due to the lack of publicly available multispectral datasets in this topic, first, a freely accessible dataset was generated including 100 pairs of visible and thermal images of rice and weeds. In this new dataset, the segmented plants were labeled into two groups of rice and weeds. A feature vector including 15 morphological, 12 spectral, 10 textural and 11 new thermal features was extracted from segmented objects. The proposed thermal features were extracted from thermal images, while other types of features were extracted from visible/thermal pairs. Then, a genetic algorithm (GA) was used for optimized feature selection. Next, multiple late and early fusion structures at the decision level were developed and compared for weed detection purposes. Applied fusion structures include: multilayer perceptron neural network (MLP), extreme learning machines (ELM) and extreme learning machines ensembles (ELM-E) artificial neural networks. The best results were obtained by ELM with an accuracy of 98.08% in late fusion structure with GA. The best results for classifying plants in visible images were related to the morphological descriptor and the best results for thermal images belonged to the proposed descriptor.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Abbreviations

SSWM:

Site-specific weed management

FCN:

Fully convolutional networks

NGRDI:

Normalized green red difference index

GLCM:

Gray-level co-occurrence matrix

RF:

Random forest

r:

Normalized value of the R component of the image

b:

Normalized value of the B component of the image

TP:

True positive or pixels correctly segmented as foreground

TN:

True negative or pixels correctly detected as background

CFI:

Choquet fuzzy integral

DES:

Dempster Shafer theory

LBP:

Local binary pattern

ELM-E:

Extreme learning machines ensembles

µm:

Micrometre

nm:

Nanometre

FD:

Fourier descriptor

ANN:

Artificial neural networks

EA:

Evolutionary algorithms

RGB:

Red, green, and blue components of image

MPA:

Mean pixel accuracy

NDVI:

Normalized difference vegetation index

MLP:

Multilayer perceptron neural network

SVM:

Support vector machines

SPA:

Successive projection algorithm

g:

Normalized value of the G component of the image

FP:

False positive or pixels falsely segmented as foreground

FN:

True negative or pixels falsely detected as background

SFI:

Sugeno fuzzy integral

FMCDM:

Fuzzy multicriteria decision making

ELM:

Extreme learning machines

GA:

Genetic algorithm

SP:

Select pressure

mm:

Millimeters

MI:

Moment invariants

TSD:

Tensor scale descriptor

ACO:

Ant colony optimization algorithm

References

  • Ahmed, F., Al-Mamun, H. A., Bari, A. H., Hossain, E., & Kwan, P. (2012). Classification of crops and weeds from digital images: A support vector machine approach. Crop Protection, 40, 98–104.

    Article  Google Scholar 

  • Akbarzadeh, S., Paap, A., Ahderom, S., Apopei, B., & Alameh, K. (2018). Plant discrimination by support vector machine classifier based on spectral reflectance. Computers and Electronics in Agriculture, 148, 250–258.

    Article  Google Scholar 

  • Asvadi, A., Karami, M., & Baleghi, Y. (2011). Efficient object tracking using optimized K-means segmentation and radial basis function neural networks. International Journal of Information and Communication Technology, 4, 29–39.

    Google Scholar 

  • Asvadi, A., Mahdavinataj, H., Karami, M., & Baleghi, Y. (2013). Incremental discriminative color object tracking. In International symposium on artificial intelligence and signal processing (pp. 71–81). Springer.

  • Bakhshipour, A., & Jafari, A. (2018). Evaluation of support vector machine and artificial neural networks in weed detection using shape features. Computers and Electronics in Agriculture, 145, 153–160.

    Article  Google Scholar 

  • Bakhshipour, A., Jafari, A., Nassiri, S. M., & Zare, D. (2017). Weed segmentation using texture features extracted from wavelet sub-images. Biosystems Engineering, 157, 1–12.

    Article  Google Scholar 

  • Barrero, O., & Perdomo, S. A. (2018). RGB and multispectral UAV image fusion for Gramineae weed detection in rice fields. Precision Agriculture, 19, 809–822.

    Article  Google Scholar 

  • Bharati, M. H., Liu, J. J., & MacGregor, J. F. (2004). Image texture analysis: Methods and comparisons. Chemometrics and Intelligent Laboratory Systems, 72, 57–71.

    Article  CAS  Google Scholar 

  • Brown, R. B., & Noble, S. D. (2005). Site-specific weed management: Sensing requirements—What do we need to see? Weed Science, 53, 252–258.

    Article  CAS  Google Scholar 

  • Cao, J., Lin, Z., Huang, G.-B., & Liu, N. (2012). Voting based extreme learning machine. Information Sciences, 185, 66–77.

    Article  Google Scholar 

  • Chaudhuri, B., & Bhattacharya, U. (2000). Efficient training and improved performance of multilayer perceptron in pattern classification. Neurocomputing, 34, 11–27.

    Article  Google Scholar 

  • Cheng, B., & Matson, E. T. (2015). A feature-based machine learning agent for automatic rice and weed discrimination. In International conference on artificial intelligence and soft computing (pp. 517–527). Springer.

  • Cvetković, S., Stojanović, M. B., & Nikolić, S. V. (2018). Hierarchical ELM ensembles for visual descriptor fusion. Information Fusion, 41, 16–24.

    Article  Google Scholar 

  • Doustfatemeh, I., & Baleghi, Y. (2016). Comprehensive urban area extraction from multispectral medium spatial resolution remote-sensing imagery based on a novel structural feature. International Journal of Remote Sensing, 37, 4225–4242.

    Article  Google Scholar 

  • Fawakherji, M., Potena, C., Pretto, A., Bloisi, D. D., & Nardi, D. (2021). Multi-spectral image synthesis for crop/weed segmentation in precision farming. Robotics and Autonomous Systems, 146, 103861.

    Article  Google Scholar 

  • Gokulnath, C. B., & Shantharajah, S. (2019). An optimized feature selection based on genetic approach and support vector machine for heart disease. Cluster Computing, 22, 14777–14787.

    Article  Google Scholar 

  • Guijarro, M., Pajares, G., Riomoros, I., Herrera, P., Burgos-Artizzu, X., & Ribeiro, A. (2011). Automatic segmentation of relevant textures in agricultural images. Computers and Electronics in Agriculture, 75, 75–83.

    Article  Google Scholar 

  • Hamuda, E., Glavin, M., & Jones, E. (2016). A survey of image processing techniques for plant extraction and segmentation in the field. Computers and Electronics in Agriculture, 125, 184–199.

    Article  Google Scholar 

  • Herrera, P. J., Dorado, J., & Ribeiro, Á. (2014). A novel approach for weed type classification based on shape descriptors and a fuzzy decision-making method. Sensors, 14, 15304–15324.

    Article  PubMed  PubMed Central  Google Scholar 

  • Hu, M.-K. (1962). Visual pattern recognition by moment invariants. IRE Transactions on Information Theory, 8, 179–187.

    Article  Google Scholar 

  • Huang, Z., & Leng, J. (2010). Analysis of Hu's moment invariants on image scaling and rotation. In 2010 2nd international conference on computer engineering and technology (pp. 476–480). IEEE.

  • Jensen, H. G., Jacobsen, L.-B., Pedersen, S. M., & Tavella, E. (2012). Socioeconomic impact of widespread adoption of precision farming and controlled traffic systems in Denmark. Precision Agriculture, 13, 661–677.

    Article  Google Scholar 

  • Kakooei, M., & Baleghi, Y. (2020). A two-level fusion for building irregularity detection in post-disaster VHR oblique images. Earth Science Informatics, 13, 459–477.

    Article  Google Scholar 

  • Liu, K., Li, Y., Xu, N., & Natarajan, P. (2018). Learn to combine modalities in multimodal deep learning. Preprint at http://arXiv.org/1805.11730

  • López Granados, F. (2011). Weed detection for site-specific weed management: Mapping and real-time approaches. Weed Research, 51, 1–11.

    Article  Google Scholar 

  • Ma, X., Deng, X., Qi, L., Jiang, Y., Li, H., Wang, Y., et al. (2019). Fully convolutional network for rice seedling and weed image segmentation at the seedling stage in paddy fields. PLoS ONE, 14, e0215676.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Mitchell, H. B. (2010). Image fusion: Theories, techniques and applications. Springer.

    Book  Google Scholar 

  • Montalvo, M., Guerrero, J. M., Romeo, J., Emmi, L., Guijarro, M., & Pajares, G. (2013). Automatic expert system for weeds/crops identification in images from maize fields. Expert Systems with Applications, 40, 75–82.

    Article  Google Scholar 

  • Nikbakhsh, N., & Baleghi, Y. (2019). A new fast method of image segmentation fusion using maximum mutual information. In 2019 27th Iranian conference on electrical engineering (ICEE) (pp. 1584–1588). IEEE.

  • Nikbakhsh, N., Baleghi, Y., & Agahi, H. (2019). Maximum mutual information and Tsallis entropy for unsupervised segmentation of tree leaves in natural scenes. Computers and Electronics in Agriculture, 162, 440–449.

    Article  Google Scholar 

  • Nikbakhsh, N., Baleghi, Y., & Agahi, H. (2020b). A novel approach for unsupervised image segmentation fusion of plant leaves based on G-mutual information. Machine Vision and Applications, 32, 1–12.

    Google Scholar 

  • Nikbakhsh, N., Baleghi Damavandi, Y., & Agahi, H. (2020a). Plant classification in images of natural scenes using segmentations fusion. International Journal of Engineering, 33, 1743–1750.

    Google Scholar 

  • Ojala, T., Pietikainen, M., & Maenpaa, T. (2002). Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24, 971–987.

    Article  Google Scholar 

  • Pandeya, Y. R., & Lee, J. (2021). Deep learning-based late fusion of multimodal information for emotion classification of music video. Multimedia Tools and Applications, 80, 2887–2905.

    Article  Google Scholar 

  • Rodrigo, M., Oturan, N., & Oturan, M. A. (2014). Electrochemically assisted remediation of pesticides in soils and water: A review. Chemical Reviews, 114, 8720–8745.

    Article  CAS  PubMed  Google Scholar 

  • Shirzadifar, A., Bajwa, S., Nowatzki, J., & Shojaeiarani, J. (2020). Development of spectral indices for identifying glyphosate-resistant weeds. Computers and Electronics in Agriculture, 170, 105276.

    Article  Google Scholar 

  • Slaughter, D., Giles, D., & Downey, D. (2008). Autonomous robotic weed control systems: A review. Computers and Electronics in Agriculture, 61, 63–78.

    Article  Google Scholar 

  • Talbi, E. G., Basseur, M., Nebro, A. J., & Alba, E. (2012). Multi-objective optimization using metaheuristics: Non-standard algorithms. International Transactions in Operational Research, 19, 283–305.

    Article  Google Scholar 

  • Tang, J., Wang, D., Zhang, Z., He, L., Xin, J., & Xu, Y. (2017). Weed identification based on K-means feature learning combined with convolutional neural network. Computers and Electronics in Agriculture, 135, 63–70.

    Article  Google Scholar 

  • ul Hussain, S., & Triggs, B. (2012). Visual recognition using local quantized patterns. In European conference on computer vision (pp. 716–729). Springer.

  • Wang, A., Zhang, W., & Wei, X. (2019). A review on weed detection using ground-based machine vision and image processing techniques. Computers and Electronics in Agriculture, 158, 226–240.

    Article  Google Scholar 

  • Yousefi, E., Baleghi, Y., & Sakhaei, S. M. (2017). Rotation invariant wavelet descriptors, a new set of features to enhance plant leaves classification. Computers and Electronics in Agriculture, 140, 70–76.

    Article  Google Scholar 

  • Yu, J., Sharpe, S. M., Schumann, A. W., & Boyd, N. S. (2019). Deep learning for image-based weed detection in turfgrass. European Journal of Agronomy, 104, 78–84.

    Article  Google Scholar 

  • Zamani, S. A., & Baleghi, Y. Visible-thermal database of rice field. Mendeley Data, Version 3. Retrieved July 26, 2022, from https://data.mendeley.com/datasets/9xg52j8tmw/3

  • Zhang, J., Song, F., & Tang, J. (2014). Identification of crop weed based on image texture features. Moment, 67(64), 1488.

    Google Scholar 

  • Zhang, Y., Gao, J., Cen, H., Lu, Y., Yu, X., He, Y., et al. (2019). Automated spectral feature extraction from hyperspectral images to differentiate weedy rice and barnyard grass from a rice crop. Computers and Electronics in Agriculture, 159, 42–49.

    Article  Google Scholar 

Download references

Acknowledgements

The authors acknowledge the funding support of Babol Noshirvani University of Technology through Grant program No. BNUT/370123/00. The authors would also like to thank the editor and anonymous reviewers for the useful and constructive comments which have significantly improved the article.

Author information

Authors and Affiliations

  1. Department of Electrical & Computer Engineering, Babol Noshirvani University of Technology, Babol, Iran

    Seyed Alireza Zamani & Yasser Baleghi

Authors
  1. Seyed Alireza Zamani
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yasser Baleghi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yasser Baleghi.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zamani, S.A., Baleghi, Y. Early/late fusion structures with optimized feature selection for weed detection using visible and thermal images of paddy fields. Precision Agric 24, 482–510 (2023). https://doi.org/10.1007/s11119-022-09954-8

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11119-022-09954-8

Keywords

Search

Navigation