Skip to main content
SpringerLink
Log in

DeepFusion: Deep Ensembles for Domain Independent System Fusion

  • Conference paper
  • First Online:
MultiMedia Modeling (MMM 2021)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 12572))

Included in the following conference series:

Abstract

While ensemble systems and late fusion mechanisms have proven their effectiveness by achieving state-of-the-art results in various computer vision tasks, current approaches are not exploiting the power of deep neural networks as their primary ensembling algorithm, but only as inducers, i.e., systems that are used as inputs for the primary ensembling algorithm. In this paper, we propose several deep neural network architectures as ensembling algorithms with various network configurations that use dense and attention layers, an input pre-processing algorithm, and a new type of deep neural network layer denoted the Cross-Space-Fusion layer, that further improves the overall results. Experimental validation is carried out on several data sets from various domains (emotional content classification, medical data captioning) and under various evaluation conditions (two-class regression, binary classification, and multi-label classification), proving the efficiency of DeepFusion.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://github.com/cmihaigabriel/DeepFusionSystem_v2.

  2. 2.

    http://www.multimediaeval.org/.

  3. 3.

    https://www.imageclef.org/.

  4. 4.

    http://ceur-ws.org/Vol-2283/ .

  5. 5.

    http://ceur-ws.org/Vol-2380/.

References

  1. Bengio, Y., Simard, P., Frasconi, P.: Learning long-term dependencies with gradient descent is difficult. IEEE Trans. Neural Netw. 5(2), 157–166 (1994)

    Article  Google Scholar 

  2. Breiman, L.: Bagging predictors. Mach. Learn. 24(2), 123–140 (1996). https://doi.org/10.1007/BF00058655

    Article  MATH  Google Scholar 

  3. Breiman, L.: Random forests. Mach. Learn. 45(1), 5–32 (2001)

    Article  Google Scholar 

  4. Chen, T., Guestrin, C.: Xgboost: a scalable tree boosting system. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 785–794. ACM (2016)

    Google Scholar 

  5. Dellandréa, E., Huigsloot, M., Chen, L., Baveye, Y., Xiao, Z., Sjöberg, M.: The mediaeval 2018 emotional impact of movies task. In: MediaEval (2018)

    Google Scholar 

  6. Deng, L., Platt, J.C.: Ensemble deep learning for speech recognition. In: Fifteenth Annual Conference of the International Speech Communication Association (2014)

    Google Scholar 

  7. Feichtenhofer, C., Pinz, A., Zisserman, A.: Convolutional two-stream network fusion for video action recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1933–1941 (2016)

    Google Scholar 

  8. Freund, Y., Schapire, R., Abe, N.: A short introduction to boosting. J. Jpn. Soc. Artif. Intell. 14(771–780), 1612 (1999)

    Google Scholar 

  9. Freund, Y., Schapire, R.E.: A desicion-theoretic generalization of on-line learning and an application to boosting. In: Vitányi, P. (ed.) EuroCOLT 1995. LNCS, vol. 904, pp. 23–37. Springer, Heidelberg (1995). https://doi.org/10.1007/3-540-59119-2_166

    Chapter  Google Scholar 

  10. Friedman, J.H.: Greedy function approximation: a gradient boosting machine. Ann. Stat. 29, 1189–1232 (2001)

    Article  MathSciNet  Google Scholar 

  11. Gomes, H.M., Barddal, J.P., Enembreck, F., Bifet, A.: A survey on ensemble learning for data stream classification. ACM Comput. Surv. 50(2), 1–36 (2017)

    Article  Google Scholar 

  12. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

  13. Kittler, J., Hatef, M., Duin, R.P., Matas, J.: On combining classifiers. IEEE Trans. Pattern Anal. Mach. Intell. 20(3), 226–239 (1998)

    Article  Google Scholar 

  14. Kontschieder, P., Fiterau, M., Criminisi, A., Rota Bulo, S.: Deep neural decision forests. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1467–1475 (2015)

    Google Scholar 

  15. Kougia, V., Pavlopoulos, J., Androutsopoulos, I.: AUEB NLP group at imageclefmed caption 2019. In: CEUR Workshop Proceedings, CLEF2019, pp. 09–12 (2019)

    Google Scholar 

  16. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097–1105 (2012)

    Google Scholar 

  17. Kuncheva, L.I., Whitaker, C.J.: Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy. Mach. Learn. 51(2), 181–207 (2003)

    Article  Google Scholar 

  18. Li, X., Huo, Y., Jin, Q., Xu, J.: Detecting violence in video using subclasses. In: 2016 ACM Conference on Multimedia Conference, pp. 586–590 (2016)

    Google Scholar 

  19. Liu, L., et al.: Deep neural network ensembles against deception: Ensemble diversity, accuracy and robustness. arXiv preprint arXiv:1908.11091 (2019)

  20. Pelka, O., Friedrich, C.M., García Seco de Herrera, A., Müller, H.: Overview of the imageclefmed 2019 concept detection task. CLEF working notes, CEUR (2019)

    Google Scholar 

  21. Pelka, O., Koitka, S., Rückert, J., Nensa, F., Friedrich, C.M.: Radiology Objects in COntext (ROCO): a multimodal image dataset. In: Stoyanov, D., et al. (eds.) LABELS/CVII/STENT -2018. LNCS, vol. 11043, pp. 180–189. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01364-6_20

    Chapter  Google Scholar 

  22. Rätsch, G., Onoda, T., Müller, K.R.: Soft margins for AdaBoost. Mach. Learn. 42(3), 287–320 (2001). https://doi.org/10.1023/A:1007618119488

    Article  MATH  Google Scholar 

  23. Roberts, R.J.: Pubmed central: The genbank of the published literature (2001)

    Google Scholar 

  24. Sagi, O., Rokach, L.: Ensemble learning: a survey. Wiley Interdisc. Rev. Data Min Kowl. Discov. 8(4), e1249 (2018)

    Google Scholar 

  25. Sun, J.J., Liu, T., Prasad, G.: GLA in mediaeval 2018 emotional impact of movies task. In: Proceedings of the MediaEval 2018 Workshop (2018)

    Google Scholar 

  26. Wolpert, D.H.: The supervised learning no-free-lunch theorems. In: Roy, R.K., Köppen, M., Ovaska, S., Furuhashi, T. (eds.) Soft Computing and Industry, pp. 25–42. Springer, London (2002). https://doi.org/10.1007/978-1-4471-0123-9_3

  27. Yi, Y., Wang, H., Li, Q.: CNN features for emotional impact of movies task. In: Proceedings of MediaEval 2018 Workshop (2018)

    Google Scholar 

  28. Zhou, Z.H., Feng, J.: Deep forest: towards an alternative to deep neural networks. In: Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI-17, pp. 3553–3559 (2017)

    Google Scholar 

Download references

Acknowledgements

This work was funded under project SMARTRetail, agreement 8PTE/2020, grant PN-III-P2-2.1-PTE-2019-0055, Ministry of Innovation and Research, UEFISCDI and AI4Media “A European Excellence Centre for Media, Society and Democracy”, grant #951911, H2020 ICT-48-2020.

Author information

Authors and Affiliations

  1. University “Politehnica” of Bucharest, Bucharest, Romania

    Mihai Gabriel Constantin, Liviu-Daniel Ştefan & Bogdan Ionescu

Authors
  1. Mihai Gabriel Constantin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Liviu-Daniel Ştefan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Bogdan Ionescu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mihai Gabriel Constantin .

Editor information

Editors and Affiliations

  1. Charles University, Prague, Czech Republic

    Jakub Lokoč

  2. Charles University, Prague, Czech Republic

    Tomáš Skopal

  3. Klagenfurt University, Klagenfurt, Austria

    Klaus Schoeffmann

  4. CERTH-ITI, Thessaloniki, Greece

    Vasileios Mezaris

  5. Renmin University of China, Beijing, China

    Xirong Li

  6. CERTH-ITI, Thessaloniki, Greece

    Stefanos Vrochidis

  7. Queen Mary University of London, London, UK

    Ioannis Patras

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Constantin, M.G., Ştefan, LD., Ionescu, B. (2021). DeepFusion: Deep Ensembles for Domain Independent System Fusion. In: Lokoč, J., et al. MultiMedia Modeling. MMM 2021. Lecture Notes in Computer Science(), vol 12572. Springer, Cham. https://doi.org/10.1007/978-3-030-67832-6_20

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-67832-6_20

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-67831-9

  • Online ISBN: 978-3-030-67832-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation