Abstract
The development of social networking services (SNSs) revealed a surge in image sharing. The sharing mode of multi-page photo collage (MPC), which posts several image collages at a time, can often be observed on many social network platforms, which enables uploading images and arrangement in a logical order. This study focuses on the construction of MPC for an image collection and its formulation as an issue of joint optimization, which involves not only the arrangement in a single collage but also the arrangement among different collages. Novel balance-aware measurements, which merge graphic features and psychological achievements, are introduced. Non-dominated sorting genetic algorithm is adopted to optimize the MPC guided by the measurements. Experiments demonstrate that the proposed method can lead to diverse, visually pleasant, and logically clear MPC results, which are comparable to manually designed MPC results.
Article PDF
Similar content being viewed by others
References
Yu, Z. Q.; Lu, L.; Guo, Y. W.; Fan, R. F.; Liu, M. M.; Wang, W. P. Content-aware photo collage using circle packing. IEEE Transactions on Visualization and Computer Graphics Vol. 20, No. 2, 182–195, 2014.
Liu, L. J.; Zhang, H. J.; Jing, G. M.; Guo, Y. W.; Chen, Z. G.; Wang, W. P. Correlation-preserving photo collage. IEEE Transactions on Visualization and Computer Graphics Vol. 24, No. 6, 1956–1968, 2018.
Liang, Y.; Wang, X. T.; Zhang, S. H.; Hu, S. M.; Liu, S. X. PhotoRecomposer: Interactive photo recomposition by cropping. IEEE Transactions on Visualization and Computer Graphics Vol. 24, No. 10, 2728–2742, 2018.
Pan, X. J.; Tang, F.; Dong, W. M.; Ma, C. Y.; Meng, Y. P.; Huang, F. Y.; Lee, T. Y.; Xu, C. S. Content-based visual summarization for image collections. IEEE Transactions on Visualization and Computer Graphics Vol. 27, No. 4, 2298–2312, 2021.
Wagemans, J.; Elder, J. H.; Kubovy, M.; Palmer, S. E.; Peterson, M. A.; Singh, M.; von der Heydt, R. A century of Gestalt psychology in visual perception: I. Perceptual grouping and figure-ground organization. Psychological Bulletin Vol. 138, No. 6, 1172–1217, 2012.
Hübner, R.; Fillinger, M. G. Comparison of objective measures for predicting perceptual balance and visual aesthetic preference. Frontiers in Psychology Vol. 7, 335, 2016.
Hüubner, R.; Fillinger, M. G. Perceptual balance, stability, and aesthetic appreciation: Their relations depend on the picture type. i-Perception Vol. 10, No. 3, 1–17, 2019.
Wilson, A.; Chatterjee, A. The assessment of preference for balance: Introducing a new test. Empirical Studies of the Arts Vol. 23, No. 2, 165–180, 2005.
Coello Coello, C.; Lamont, G. B.; van Veldhuizen, D. A. Evolutionary Algorithms for Solving MultiObjective Problems. Springer US, 2007.
Tan, L.; Song, Y. Q.; Liu, S. X.; Xie, L. X. ImageHive: Interactive content-aware image summarization. IEEE Computer Graphics and Applications Vol. 32, No. 1, 46–55, 2012.
Cao, Y.; Chan, A. B.; Lau, R. W. H. Automatic stylistic manga layout. ACM Transactions on Graphics Vol. 31, No. 6, Article No. 141, 2012.
Cao, Y.; Lau, R. W. H.; Chan, A. B. Look over here: Attention-directing composition of manga elements. ACM Transactions on Graphics Vol. 33, No. 4, Article No. 94, 2014.
Chen, T.; Lu, A. D.; Hu, S. M. Visual storylines: Semantic visualization of movie sequence. Computers & Graphics Vol. 36, No. 4, 241–249, 2012.
Jing, G. M.; Hu, Y. T.; Guo, Y. W.; Yu, Y. Z.; Wang, W. P. Content-aware Video2Comics with manga-style layout. IEEE Transactions on Multimedia Vol. 17, No. 12, 2122–2133, 2015.
Han, X. T.; Zhang, C. Y.; Lin, W. Y.; Xu, M. L.; Sheng, B.; Mei, T. Tree-based visualization and optimization for image collection. IEEE Transactions on Cybernetics Vol. 46, No. 6, 1286–1300, 2016.
Wu, Z. P.; Aizawa, K. Very fast generation of content-preserved photo collage under canvas size constraint. Multimedia Tools and Applications Vol. 75, No. 4, 1813–1841, 2016.
Zheng, X. R.; Qiao, X. T.; Cao, Y.; Lau, R. W. H. Content-aware generative modeling of graphic design layouts. ACM Transactions on Graphics Vol. 38, No. 4, Article No. 133, 2019.
Gan, Y.; Zhang, Y.; Sun, Z. X.; Zhang, H. Qualitative photo collage by quartet analysis and active learning. Computers & Graphics Vol. 88, 35–44, 2020.
Kong, S.; Shen, X. H.; Lin, Z.; Mech, R.; Fowlkes, C. Photo aesthetics ranking network with attributes and content adaptation. In: Computer Vision — ECCV 2016. Lecture Notes in Computer Science, Vol. 9905. Leibe, B.; Matas, J.; Sebe, N.; Welling, M. Eds. Springer Cham, 662–679, 2016.
Song, Y.; Tang, F.; Dong, W. M.; Huang, F. Y.; Xu, C. S. Balance-based photo posting. In: Proceedings of the SIGGRAPH Asia 2019 Posters, Article No. 20, 2019.
Deb, K. Multi-Objective Optimization Using Evolutionary Algorithms. New York: Wiley, 2001.
Deb, K.; Pratap, A.; Agarwal, S.; Meyarivan, T. A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Transactions on Evolutionary Computation Vol. 6, No. 2, 182–197, 2002.
Li, D.; Zhong, J. Dimensionally aware multi-objective genetic programming for automatic crowd behavior modeling. ACM Transactions on Modeling and Computer Simulation Vol. 30, No. 3, Article No. 19, 2020.
Lee, C.; Shin, S.; Issa, R. R. A. Multi-objective optimization of a free-form surface based on generative designs. In: Proceedings of the 18th International Conference on Computing in Civil and Building Engineering. Lecture Notes in Civil Engineering, Vol. 98. Toledo Santos, E.; Scheer, S. Eds. Springer Cham, 1252–1261, 2021.
Rejeesh, M. R.; Thejaswini, P. MOTF: Multi-objective Optimal Trilateral Filtering based partial moving frame algorithm for image denoising. Multimedia Tools and Applications Vol. 79, Nos. 37–38, 28411–28430, 2020.
Su, J.; Yin, H. HypervolGAN: An efficient approach for GAN with multi-objective training function. arXiv preprint arXiv:2006.15228, 2020.
Chen, Z. B.; Lin, J. X.; Liao, N.; Chen, C. W. Full reference quality assessment for image retargeting based on natural scene statistics modeling and Bidirectional saliency similarity. IEEE Transactions on Image Processing Vol. 26, No. 11, 5138–5148, 2017.
Zhang, Y. B.; Lin, W. S.; Li, Q. H.; Cheng, W. T.; Zhang, X. F. Multiple-level feature-based measure for retargeted image quality. IEEE Transactions on Image Processing Vol. 27, No. 1, 451–463, 2018.
Ma, L.; Xu, L.; Zhang, Y. C.; Yan, Y. H.; Ngan, K. N. No-reference retargeted image quality assessment based on pairwise rank learning. IEEE Transactions on Multimedia Vol. 18, No. 11, 2228–2237, 2016.
Liang, Y.; Liu, Y. J.; Gutierrez, D. Objective quality prediction of image retargeting algorithms. IEEE Transactions on Visualization and Computer Graphics Vol. 23, No. 2, 1099–1110, 2017.
Sheng, K. K.; Dong, W. M.; Ma, C. Y.; Mei, X.; Huang, F. Y.; Hu, B. G. Attention-based multipatch aggregation for image aesthetic assessment. In: Proceedings of the 26th ACM International Conference on Multimedia, 879–886, 2018.
Sheng, K. K.; Dong, W. M.; Huang, H. B.; Chai, M. L.; Zhang, Y.; Ma, C. Y.; Hu, B.-G. Learning to assess visual aesthetics of food images. Computational Visual Media Vol. 7, No. 1, 139–152, 2021.
Xu, M. N.; Zhong, J. X.; Ren, Y. R.; Liu, S.; Li, G. Context-aware attention network for predicting image aesthetic subjectivity. In: Proceedings of the 28th ACM International Conference on Multimedia, 798–806, 2020.
Simonyan, K.; Zisserman, A. Very deep convolutional networks for large-scale image recognition. In: Proceedings of the 3rd International Conference on Learning Representations, 2015.
Thömmes, K.; Hübner, R. Instagram likes for architectural photos can be predicted by quantitative balance measures and curvature. Frontiers in Psychology Vol. 9, 1050, 2018.
Acknowledgements
We thank the anonymous reviewers for their valuable comments. We thank Dr. Xiaowen Huang for providing test images. This work was supported by National Key R&D Program of China under No. 2020AAA0106200, and by National Natural Science Foundation of China under Nos. 61832016 and U20B2070.
Author information
Authors and Affiliations
Corresponding author
Additional information
Yu Song received her M.Sc. degree in automation from Tianjin University in 2017. She is currently working towards a Ph.D. degree in the National Laboratory of Pattern Recognition (NLPR), Institute of Automation, Chinese Academy of Sciences. Her research interests include image collage, image retargeting, and machine learning.
Fan Tang is an assistant professor in the School of Artificial Intelligence, Jilin University. He received his B.Sc. degree in computer science from North China Electric Power University in 2013 and his Ph.D. degree from NLPR in 2019. His research interests include computer graphics, computer vision, and machine learning.
Weiming Dong is a professor in the Sino-European Lab of Computer Science, Automation and Applied Mathematics (LIAMA) and NLPR. He received his B.Sc. and M.Sc. degrees in computer science in 2001 and 2004, both from Tsinghua University, China. He received his Ph.D. degree in computer science from the University of Lorraine, France, in 2007. His research interests include computational visual media and computational creativity. He is a member of the ACM and IEEE.
Changsheng Xu is a professor in NLPR and executive director of China-Singapore Institute of Digital Media. His research interests include multimedia content analysis, indexing and retrieval, pattern recognition, and computer vision. He has 30 granted or pending patents and published over 200 papers. He is an associate editor of IEEE Trans. on Multimedia, ACM Trans. on Multimedia Computing, Communications and Applications, and ACM/Springer Multimedia Systems Journal. He is an IEEE Fellow, IAPR Fellow, and ACM Distinguished Scientist.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.
The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.
To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
Other papers from this open access journal are available free of charge from http://www.springer.com/journal/41095. To submit a manuscript, please go to https://www.editorialmanager.com/cvmj.
About this article
Cite this article
Song, Y., Tang, F., Dong, W. et al. Non-dominated sorting based multi-page photo collage. Comp. Visual Media 8, 199–212 (2022). https://doi.org/10.1007/s41095-021-0221-0
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s41095-021-0221-0