Skip to main content
SpringerLink
Log in
Book cover

International Conference on Multimedia Modeling

MMM 2022: MultiMedia Modeling pp 84–96Cite as

ILMICA - Interactive Learning Model of Image Collage Assessment: A Transfer Learning Approach for Aesthetic Principles

  • 1861 Accesses

Part of the Lecture Notes in Computer Science book series (LNCS,volume 13142)

Abstract

The beauty of moments can be expressed in many ways. One of them is the image collage which captures events and expresses emotions. Nowadays there is a large number of digital images. Aesthetic analyses of image collages are rarely performed due to their complexity and time-consuming nature. For this reason, this is an important issue that has to be addressed. In this paper, we propose an interactive learning model for image collage assessment. It consists of two components: A pre-trained convolutional neural network with built-in knowledge about aesthetics obtained from single image analysis, and an “Interactive Transfer Learning” component specialized in collage aesthetics which is adaptable via Active Learning. We present a mixed method study in which rules for software-based collage generation are identified and a dataset of automatically generated collages representative of the rules is created. ILMICA’s performance is analyzed by a user survey. It is found that the knowledge transfer from single image assessment to collage assessment works: ILMICA can assess collage aesthetics based on predefined rules, thereby demonstrating the system’s ability to learn. Thus, this process can alleviate the end user and simplify aesthetic collage evaluations.

Keywords

  • Multimedia
  • Aesthetic collage evaluation
  • User-centered design
  • Convolutional neural networks
  • Transfer and active learning

This is a preview of subscription content, access via your institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   79.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

References

  1. Abdenebaoui, L., Meyer, B., Bruns, A., Boll, S.: UNNA: a unified neural network for aesthetic assessment. In: 2018 International Conference on Content-Based Multimedia Indexing (CBMI), La Rochelle, pp. 1–6. IEEE (2018). https://doi.org/10.1109/CBMI.2018.8516273

  2. Atkins, C.B.: Blocked recursive image composition. In: Proceeding of the 16th ACM International Conference on Multimedia - MM 2008, Vancouver, British Columbia, Canada, pp. 821–824. ACM Press (2008). https://doi.org/10.1145/1459359.1459496

  3. Bossard, L., Guillaumin, M., Van Gool, L.: Event recognition in photo collections with a stopwatch HMM. In: 2013 IEEE International Conference on Computer Vision, Sydney, Australia, pp. 1193–1200. IEEE (2013). https://doi.org/10.1109/ICCV.2013.151

  4. Ceroni, A., Solachidis, V., Niederée, C., Papadopoulou, O., Kanhabua, N., Mezaris, V.: To keep or not to keep: an expectation-oriented photo selection method for personal photo collections. In: Proceedings of the 5th ACM on International Conference on Multimedia Retrieval, Shanghai, China, pp. 187–194. ACM Press (2015). https://doi.org/10.1145/2671188.2749372

  5. Datta, R., Joshi, D., Li, J., Wang, J.Z.: Studying aesthetics in photographic images using a computational approach. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3953, pp. 288–301. Springer, Heidelberg (2006). https://doi.org/10.1007/11744078_23

    CrossRef  Google Scholar 

  6. Deng, Y., Loy, C.C., Tang, X.: Image aesthetic assessment: an experimental survey. IEEE Signal Process. Mag. 34(4), 80–106 (2017). https://doi.org/10.1109/MSP.2017.2696576

    CrossRef  Google Scholar 

  7. Diakopoulos, N., Essa, I.: Mediating photo collage authoring. In: Proceedings of the 18th Annual ACM Symposium on User Interface Software and Technology, Seattle, WA, USA, pp. 183–186. ACM Press (2005)

    Google Scholar 

  8. Fogarty, J., Forlizzi, J., Hudson, S.E.: Aesthetic information collages: generating decorative displays that contain information. In: Proceedings of the 14th Annual ACM Symposium on User Interface Software and Technology, Orlando, Florida, pp. 141–150. ACM Press (2001). https://doi.org/10.1145/502348.502369

  9. Girgensohn, A., Chiu, P.: Stained glass photo collages. In: IEEE International Conference on Image Processing, vol. 2, pp. 871–874 (2004)

    Google Scholar 

  10. Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press, Cambridge (2016)

    MATH  Google Scholar 

  11. Inkpen, K., Chancellor, S., De Choudhury, M., Veale, M., Baumer, E.P.S.: Where is the human?: bridging the gap between AI and HCI. In: Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems, Glasgow, Scotland, UK, pp. 1–9, May 2019. https://doi.org/10.1145/3290607.3299002

  12. ISO9241: Ergonomics of human-system interaction - Part 210: Human-centred design for interactive systems (ISO 9241-210:2019) (2019)

    Google Scholar 

  13. Jendryschik, M.: DIN EN ISO 9241-210 konkretisiert user experience - Gesamtbetrachtung. iX Magazin für Professionelle Informationstechnik 7, 108–111 (2020)

    Google Scholar 

  14. Kang, C., Valenzise, G., Dufaux, F.: EVA: an explainable visual aesthetics dataset. In: Joint Workshop on Aesthetic and Technical Quality Assessment of Multimedia and Media Analytics for Societal Trends, Seattle, WA, USA, pp. 5–13, October 2020. https://doi.org/10.1145/3423268.3423590

  15. Kong, S., Shen, X., Lin, Z., Mech, R., Fowlkes, C.: Photo aesthetics ranking network with attributes and content adaptation. arXiv:1606.01621, pp. 662–679, July 2016

  16. Liu, T., Wang, J., Sun, J., Zheng, N., Tang, X., Shum, H.Y.: Picture collage. IEEE Trans. Multimedia 11(7), 1225–1239 (2009). https://doi.org/10.1109/TMM.2009.2030741

    CrossRef  Google Scholar 

  17. Lu, X., Lin, Z., Jin, H., Yang, J., Wang, J.Z.: RAPID: rating pictorial aesthetics using deep learning. In: Proceedings of the ACM International Conference on Multimedia - MM 2014, Orlando, Florida, USA, pp. 457–466. ACM Press (2014). https://doi.org/10.1145/2647868.2654927

  18. Malu, G., Bapi, R.S., Indurkhya, B.: Learning photography aesthetics with deep CNNs. arXiv preprint arXiv:1707.03981, July 2017

  19. Ren, J., Shen, X., Lin, Z., Mech, R., Foran, D.J.: Personalized image aesthetics. In: 2017 IEEE International Conference on Computer Vision (ICCV), Venice, pp. 638–647. IEEE, October 2017. https://doi.org/10.1109/ICCV.2017.76

  20. Rother, C., Bordeaux, L., Hamadi, Y., Blake, A.: Autocollage. ACM Trans. Graph. 25(3), 847–852 (2006). https://doi.org/10.1145/1141911.1141965

    CrossRef  Google Scholar 

  21. Sandhaus, P., Rabbath, M., Boll, S.: Employing aesthetic principles for automatic photo book layout. In: Lee, K.-T., Tsai, W.-H., Liao, H.-Y.M., Chen, T., Hsieh, J.-W., Tseng, C.-C. (eds.) MMM 2011. LNCS, vol. 6523, pp. 84–95. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-17832-0_9

    CrossRef  Google Scholar 

  22. Settles, B.: Active learning. Synth. Lect. Artif. Intell. Mach. Learn. 6(1), 1–114 (2012)

    MathSciNet  MATH  Google Scholar 

  23. Wang, W., Zhao, M., Wang, L., Huang, J., Cai, C., Xu, X.: A multi-scene deep learning model for image aesthetic evaluation. Signal Process. Image Commun. 47, 511–518 (2016). https://doi.org/10.1016/j.image.2016.05.009

    CrossRef  Google Scholar 

  24. Wu, Z., Aizawa, K.: PicWall: photo collage on-the-fly. In: Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, Kaohsiung, Taiwan, pp. 1–10. IEEE, October 2013. https://doi.org/10.1109/APSIPA.2013.6694305

  25. Wu, Z., Aizawa, K.: Very fast generation of content-preserved photo collage under canvas size constraint. Multimedia Tools Appl. 75(4), 1813–1841 (2014). https://doi.org/10.1007/s11042-014-2375-6

    CrossRef  Google Scholar 

  26. Xiao, J., Zhang, X., Cheatle, P., Gao, Y., Atkins, C.B.: Mixed-initiative photo collage authoring. In: Proceeding of the 16th ACM International Conference on Multimedia, Vancouver, British Columbia, Canada, pp. 509–518. ACM Press (2008). https://doi.org/10.1145/1459359.1459427

  27. Yoo, D., Kweon, I.S.: Learning loss for active learning. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, pp. 93–102. IEEE, June 2019. https://doi.org/10.1109/CVPR.2019.00018

Download references

Acknowledgements

We thank the reviewers for their valuable suggestions and comments. We appreciate the time and effort they invested in the review.

Author information

Authors and Affiliations

  1. OFFIS - Institute for Information Technology, Oldenburg, Germany

    Ani Withöft & Larbi Abdenebaoui

  2. Carl von Ossietzky University, Oldenburg, Germany

    Susanne Boll

Authors
  1. Ani Withöft
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Larbi Abdenebaoui
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Susanne Boll
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ani Withöft .

Editor information

Editors and Affiliations

  1. IT University of Copenhagen, Copenhagen, Denmark

    Björn Þór Jónsson

  2. Dublin City University, Dublin, Ireland

    Dr. Cathal Gurrin

  3. University of Science, VNU-HCM, Ho Chi Minh City, Vietnam

    Minh-Triet Tran

  4. University of Bergen, Bergen, Norway

    Duc-Tien Dang-Nguyen

  5. National Tsing Hua University, Hsinchu, Taiwan

    Anita Min-Chun Hu

  6. Hanoi University of Science and Technology, Hanoi, Vietnam

    Binh Huynh Thi Thanh

  7. Median Technologies, Valbonne, France

    Dr. Benoit Huet

Rights and permissions

Reprints and Permissions

Copyright information

© 2022 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Withöft, A., Abdenebaoui, L., Boll, S. (2022). ILMICA - Interactive Learning Model of Image Collage Assessment: A Transfer Learning Approach for Aesthetic Principles. In: Þór Jónsson, B., et al. MultiMedia Modeling. MMM 2022. Lecture Notes in Computer Science, vol 13142. Springer, Cham. https://doi.org/10.1007/978-3-030-98355-0_8

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-98355-0_8

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-98354-3

  • Online ISBN: 978-3-030-98355-0

  • eBook Packages: Computer ScienceComputer Science (R0)

search

Navigation