Skip to main content

Early and Late Fusion of Multiple Modalities in Sentinel Imagery and Social Media Retrieval

  • Conference paper
  • First Online:
Pattern Recognition. ICPR International Workshops and Challenges (ICPR 2021)

Abstract

Discovering potential concepts and events by analyzing Earth Observation (EO) data may be supported by fusing other distributed data sources such as non-EO data, for instance, in-situ citizen observations from social media. The retrieval of relevant information based on a target query or event is critical for operational purposes, for example, to monitor flood events in urban areas, and crop monitoring for food security scenarios. To that end, we propose an early-fusion (low-level features) and late-fusion (high-level concepts) mechanism that combines the results of two EU-funded projects for information retrieval in Sentinel imagery and social media data sources. In the early fusion part, the model is based on active learning that effectively merges Sentinel-1 and Sentinel-2 bands, and assists users to extract patterns. On the other hand, the late fusion mechanism exploits the context of other geo-referenced data such as social media retrieval, to further enrich the list of retrieved Sentinel image patches. Quantitative and qualitative results show the effectiveness of our proposed approach.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    https://www.candela-h2020.eu/.

  2. 2.

    https://lps19.esa.int/.

  3. 3.

    https://eopen-project.eu/.

  4. 4.

    https://creodias.eu/.

  5. 5.

    https://www.monetdb.org/.

  6. 6.

    http://wiki.services.eoportal.org/tiki-index.php?page=EOLib.

  7. 7.

    https://lucerne.apache.org/.

  8. 8.

    https://www.mongodb.com/.

  9. 9.

    https://developer.twitter.com/en/docs/twitter-api.

  10. 10.

    http://clc.gios.gov.pl/images/clc_ryciny/clc_classes.png.

References

  1. Andreadis, S., Bakratsas, M., Giannakeris, P., et al.: Multimedia analysis techniques for flood detection using images, articles and satellite imagery. In: Working Notes Proceedings of the MediaEval 2019 Workshop, Sophia Antipolis, France, 27–30 October 2019. CEUR Workshop Proceedings, vol. 2670. CEUR-WS.org (2019)

    Google Scholar 

  2. Atrey, P.K., Hossain, M.A., El Saddik, A., Kankanhalli, M.S.: Multimodal fusion for multimedia analysis: a survey. Multimed. Syst. 16(6), 345–379 (2010)

    Article  Google Scholar 

  3. Blanchart, P., Ferecatu, M., Cui, S., Datcu, M., et al.: Pattern retrieval in large image databases using multiscale coarse-to-fine cascaded active learning. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 7(4), 1127–1141 (2014)

    Article  Google Scholar 

  4. Chaabouni-Chouayakh, H., Datcu, M.: Backscattering and statistical information fusion for urban area mapping using TerraSAR-X data. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 3(4), 718–730 (2010)

    Article  Google Scholar 

  5. Cui, S., Dumitru, C.O., Datcu, M.: Ratio-detector-based feature extraction for very high resolution SAR image patch indexing. IEEE Geosci. Remote Sens. Lett. 10(5), 1175–1179 (2013)

    Article  Google Scholar 

  6. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805 (2018)

  7. Dumitru, C.O., Schwarz, G., Datcu, M.: Monitoring of coastal environments using data mining. In: Knowledge Extraction and Semantic Annotation (KESA 2018), pp. 34–39, April 2018

    Google Scholar 

  8. Gialampoukidis, I., Moumtzidou, A., Bakratsas, M., Vrochidis, S., Kompatsiaris, I.: A multimodal tensor-based late fusion approach for satellite image search in sentinel 2 images. In: Lokoč, J., et al. (eds.) MMM 2021. LNCS, vol. 12573, pp. 294–306. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-67835-7_25

    Chapter  Google Scholar 

  9. Gialampoukidis, I., Moumtzidou, A., Liparas, D., et al.: Multimedia retrieval based on non-linear graph-based fusion and partial least squares regression. Multimed. Tools Appl. 76(21), 22383–22403 (2017)

    Article  Google Scholar 

  10. Jegou, H., Douze, M., Schmid, C.: Product quantization for nearest neighbor search. IEEE Trans. Pattern Anal. Mach. Intell. 33(1), 117–128 (2010)

    Article  Google Scholar 

  11. Kitanovski, I., Strezoski, G., Dimitrovski, I., Madjarov, G., Loskovska, S.: Multimodal medical image retrieval system. Multimed. Tools Appl. 76(2), 2955–2978 (2016). https://doi.org/10.1007/s11042-016-3261-1

    Article  Google Scholar 

  12. Ma, X., Hovy, E.: End-to-end sequence labeling via bi-directional LSTM-CNNS-CRF. arXiv preprint arXiv:1603.01354 (2016)

  13. Mantsis, D.F., Bakratsas, M., Andreadis, S., et al.: Multimodal fusion of sentinel 1 images and social media data for snow depth estimation. IEEE Geosci. Remote Sens. Lett. (2020)

    Google Scholar 

  14. Andreadis, S., et al.: VERGE in VBS 2019. In: Kompatsiaris, I., Huet, B., Mezaris, V., Gurrin, C., Cheng, W.-H., Vrochidis, S. (eds.) MMM 2019. LNCS, vol. 11296, pp. 602–608. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-05716-9_53

    Chapter  Google Scholar 

  15. Palubinsks, G., Datcu, M.: Information fusion approach for the data classification: an example for ERS-1/2 InSAR data. Int. J. Remote Sens. 29(16), 4689–4703 (2008)

    Article  Google Scholar 

  16. Pittaras, N., Markatopoulou, F., Mezaris, V., Patras, I.: Comparison of fine-tuning and extension strategies for deep convolutional neural networks. In: Amsaleg, L., Guðmundsson, G., Gurrin, C., Jónsson, B., Satoh, S. (eds.) MMM 2017. LNCS, vol. 10132, pp. 102–114. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-51811-4_9

    Chapter  Google Scholar 

  17. Yao, W., Dumitru, C.O., Datcu, M.: D2.8 data fusion v2, deliverable of the candela project. https://www.candela-h2020.eu/content/data-fusion-v2

  18. Yao, W., Dumitru, C.O., Lorenzo, J., Datcu, M.: Data fusion on the candela cloud platform. In: European Geosciences Union (EGU) General Assembly - Big Data and Machine Learning in Geosciences, May 2020

    Google Scholar 

  19. Younessian, E., Mitamura, T., Hauptmann, A.: Multimodal knowledge-based analysis in multimedia event detection. In: Proceedings of the 2nd ACM International Conference on Multimedia Retrieval, pp. 1–8 (2012)

    Google Scholar 

Download references

Acknowledgements

This work has been supported by the EC-funded projects CANDELA (H2020-776193) and EOPEN (H2020-776019), and partly by the ASD HGF project. The content of this paper (DLR part) is mainly based on the results presented in the CANDELA Deliverable D2.8 [17].

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Stelios Andreadis .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Yao, W. et al. (2021). Early and Late Fusion of Multiple Modalities in Sentinel Imagery and Social Media Retrieval. In: Del Bimbo, A., et al. Pattern Recognition. ICPR International Workshops and Challenges. ICPR 2021. Lecture Notes in Computer Science(), vol 12667. Springer, Cham. https://doi.org/10.1007/978-3-030-68787-8_43

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-68787-8_43

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-68786-1

  • Online ISBN: 978-3-030-68787-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics