Image editing by object-aware optimal boundary searching and mixed-domain composition
When combining very different images which often contain complex objects and backgrounds, producing consistent compositions is a challenging problem requiring seamless image editing. In this paper, we propose a general approach, called object-aware image editing, to obtain consistency in structure, color, and texture in a unified way. Our approach improves upon previous gradient-domain composition in three ways. Firstly, we introduce an iterative optimization algorithm to minimize mismatches on the boundaries when the target region contains multiple objects of interest. Secondly, we propose a mixed-domain consistency metric for measuring gradients and colors, and formulate composition as a unified minimization problem that can be solved with a sparse linear system. In particular, we encode texture consistency using a patch-based approach without searching and matching. Thirdly, we adopt an object-aware approach to separately manipulate the guidance gradient fields for objects of interest and backgrounds of interest, which facilitates a variety of seamless image editing applications. Our unified method outperforms previous state-of-the-art methods in preserving global texture consistency in addition to local structure continuity.
Keywordsseamless image editing patch-based synthesis image composition mixed-domain gradient-domain composition
This work was supported in part by the National Key Research and Development Plan (Grant No. 2016YFC0801005), the National Natural Science Foundation of China (Grant Nos. 61772513 and 61402463), and the Open Foundation Project of Robot Technology Used for Special Environment Key Laboratory of Sichuan Province in China (Grant No. 16kftk01).
- Cheng, M.-M.; Zhang, F.-L.; Mitra, N. J.; Huang, X.; Hu, S.-M. RepFinder: Finding approximately repeated scene elements for image editing. ACM Transactions on Graphics Vol. 29, No. 4, Article No. 83, 2010.Google Scholar
- Barnes, C.; Zhang, F.-L.; Lou, L.; Wu, X.; Hu, S.-M. PatchTable: Efficient patch queries for large datasets and applications. ACM Transactions on Graphics Vol. 34, No. 4, Article No. 97, 2015.Google Scholar
- Efros, A. A.; Freeman, W. T. Image quilting for texture synthesis and transfer. In: Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, 341–346, 2001.Google Scholar
- Darabi, S.; Shechtman, E.; Barnes, C.; Goldman, D. B.; Sen, P. Image melding: Combining inconsistent images using patch-based synthesis. ACM Transactions on Graphics Vol. 31, No. 4, Article No. 82, 2012.Google Scholar
- Farbman, Z.; Hoffer, G.; Lipman, Y.; Cohen-Or, D.; Lischinski, D. Coordinates for instant image cloning. ACM Transactions on Graphics Vol. 28, No. 3, Article No. 67, 2009.Google Scholar
- Bhat, P.; Zitnick, C. L.; Cohen, M.; Curless, B. GradientShop: A gradient-domain optimization framework for image and video filtering. ACM Transactions on Graphics Vol. 29, No. 2, Article No. 10, 2010.Google Scholar
- Hua, M.; Bie, X.; Zhang, M.; Wang, W. Edge-aware gradient domain optimization framework for image filtering by local propagation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2838–2845, 2014.Google Scholar
- Chen, T.; Cheng, M.-M.; Tan, P.; Shamir, A.; Hu, S.-M. Sketch2Photo: Internet image montage. ACM Transactions on Graphics Vol. 28, No. 5, Article No. 124, 2009.Google Scholar
- Lee, J. H.; Choi, I.; Kim, M. H. Laplacian patchbased image synthesis. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2727–2735, 2016.Google Scholar
- Mortensen, E. N.; Barrett, W. A. Intelligent scissors for image composition. In: Proceedings of the 22nd Annual Conference on Computer Graphics and Interactive Techniques, 191–198, 1995.Google Scholar
- Krishnan, D.; Szeliski, R. Multigrid and multilevel preconditioners for computational photography. ACM Transactions on Graphics Vol. 30, No. 6, Article No. 177, 2011.Google Scholar
Open Access The articles published in this journal are distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
Other papers from this open access journal are available free of charge from http://www.springer.com/journal/41095. To submit a manuscript, please go to https://www.editorialmanager.com/cvmj.