Abstract
When combining very different images which often contain complex objects and backgrounds, producing consistent compositions is a challenging problem requiring seamless image editing. In this paper, we propose a general approach, called object-aware image editing, to obtain consistency in structure, color, and texture in a unified way. Our approach improves upon previous gradient-domain composition in three ways. Firstly, we introduce an iterative optimization algorithm to minimize mismatches on the boundaries when the target region contains multiple objects of interest. Secondly, we propose a mixed-domain consistency metric for measuring gradients and colors, and formulate composition as a unified minimization problem that can be solved with a sparse linear system. In particular, we encode texture consistency using a patch-based approach without searching and matching. Thirdly, we adopt an object-aware approach to separately manipulate the guidance gradient fields for objects of interest and backgrounds of interest, which facilitates a variety of seamless image editing applications. Our unified method outperforms previous state-of-the-art methods in preserving global texture consistency in addition to local structure continuity.
Article PDF
References
Philip, S.; Summa, B.; Tierny, J.; Bremer, P. T.; Pascucci, V. Distributed seams for gigapixel panoramas. IEEE Transactions on Visualization and Computer Graphics Vol. 21, No. 3, 350–362, 2015.
Agarwala, A.; Dontcheva, M.; Agrawala, M.; Drucker, S.; Colburn, A.; Curless, B.; Salesin, D.; Cohen, M. Interactive digital photomontage. ACM Transactions on Graphics Vol. 23, No. 3, 294–302, 2004.
Cheng, M.-M.; Zhang, F.-L.; Mitra, N. J.; Huang, X.; Hu, S.-M. RepFinder: Finding approximately repeated scene elements for image editing. ACM Transactions on Graphics Vol. 29, No. 4, Article No. 83, 2010.
Barnes, C.; Zhang, F.-L.; Lou, L.; Wu, X.; Hu, S.-M. PatchTable: Efficient patch queries for large datasets and applications. ACM Transactions on Graphics Vol. 34, No. 4, Article No. 97, 2015.
Li, J.; Tian, Y.; Huang, T. Visual saliency with statistical priors. International Journal of Computer Vision Vol. 107, No. 3, 239–253, 2014.
Li, J.; Duan, L. Y.; Chen, X.; Huang, T.; Tian, Y. Finding the secret of image saliency in the frequency domain. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 37, No. 12, 2428–2440, 2015.
Efros, A. A.; Freeman, W. T. Image quilting for texture synthesis and transfer. In: Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, 341–346, 2001.
Kwatra, V.; Schödl, A.; Essa, I.; Turk, G.; Bobick, A. Graphcut textures: Image and video synthesis using graph cuts. ACM Transactions on Graphics Vol. 22, No. 3, 277–286, 2003.
Darabi, S.; Shechtman, E.; Barnes, C.; Goldman, D. B.; Sen, P. Image melding: Combining inconsistent images using patch-based synthesis. ACM Transactions on Graphics Vol. 31, No. 4, Article No. 82, 2012.
Tao, M. W.; Johnson, M. K.; Paris, S. Error-tolerant image compositing. International Journal of Computer Vision Vol. 103, No. 2, 178–189, 2013.
Pérez, P.; Gangnet, M.; Blake, A. Poisson image editing. ACM Transactions on Graphics Vol. 22, No. 3, 313–318, 2003.
Zomet, A.; Levin, A.; Peleg, S.; Weiss, Y. Seamless image stitching by minimizing false edges. IEEE Transactions on Image Processing Vol. 15, No. 4, 969–977, 2006.
Jia, J.; Sun, J.; Tang, C.-K.; Shum, H.-Y. Drag-anddrop pasting. ACM Transactions on Graphics Vol. 25, No. 3, 631–637, 2006.
Farbman, Z.; Hoffer, G.; Lipman, Y.; Cohen-Or, D.; Lischinski, D. Coordinates for instant image cloning. ACM Transactions on Graphics Vol. 28, No. 3, Article No. 67, 2009.
Bhat, P.; Zitnick, C. L.; Cohen, M.; Curless, B. GradientShop: A gradient-domain optimization framework for image and video filtering. ACM Transactions on Graphics Vol. 29, No. 2, Article No. 10, 2010.
Li, X. Y.; Gu, Y.; Hu, S.-M.; Martin, R. R. Mixed-domain edge-aware image manipulation. IEEE Transactions on Image Processing Vol. 22, No. 5, 1915–1925, 2013.
Sadek, R.; Facciolo, G.; Arias, P.; Caselles, V. A variational model for gradient-based video editing. International Journal of Computer Vision Vol. 103, No. 1, 127–162, 2013.
Bie, X.; Wang, W.; Sun, H.; Huang, H.; Zhang, M. Intent-aware image cloning. The Visual Computer Vol. 29, Nos. 6–8, 599–608, 2013.
Hua, M.; Bie, X.; Zhang, M.; Wang, W. Edge-aware gradient domain optimization framework for image filtering by local propagation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2838–2845, 2014.
Zhang, Y.; Ling, J.; Zhang, X.; Xie, H. Image copy-and-paste with optimized gradient. The Visual Computer Vol. 30, No. 10, 1169–1178, 2014.
Ma, L.-Q.; Xu, K. Efficient manifold preserving edit propagation with adaptive neighborhood size. Computers & Graphics Vol. 38, 167–173, 2014.
Luo, S. J.; Sun, Y. T.; Shen, I. C.; Chen, B. Y.; Chuang, Y. Y. Geometrically consistent stereoscopic image editing using patch-based synthesis. IEEE Transactions on Visualization and Computer Graphics Vol. 21, No. 1, 56–67, 2015.
Chen, T.; Cheng, M.-M.; Tan, P.; Shamir, A.; Hu, S.-M. Sketch2Photo: Internet image montage. ACM Transactions on Graphics Vol. 28, No. 5, Article No. 124, 2009.
Zhang, F. L.; Wang, J.; Shechtman, E.; Zhou, Z. Y.; Shi, J. X.; Hu, S. M. PlenoPatch: Patch-based plenoptic image manipulation. IEEE Transactions on Visualization and Computer Graphics Vol. 23, No. 5, 1561–1573, 2016.
Lee, J. H.; Choi, I.; Kim, M. H. Laplacian patchbased image synthesis. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2727–2735, 2016.
Mortensen, E. N.; Barrett, W. A. Intelligent scissors for image composition. In: Proceedings of the 22nd Annual Conference on Computer Graphics and Interactive Techniques, 191–198, 1995.
Sethian, J. Level Set Methods and Fast Marching Methods: Evolving Interfaces in Computational Geometry, Fluid Mechanics, Computer Vision, and Materials Sciences. Cambridge University Press, 1999.
Rother, C.; Kolmogorov, V.; Blake, A. “GrabCut”: Interactive foreground extraction using iterated graph cuts. ACM Transactions on Graphics Vol. 23, No. 3, 309–314, 2004.
Krishnan, D.; Szeliski, R. Multigrid and multilevel preconditioners for computational photography. ACM Transactions on Graphics Vol. 30, No. 6, Article No. 177, 2011.
Mittal, A.; Soundararajan, R.; Bovik, A. C. Making a “completely blind” image quality analyzer. IEEE Signal Processing Letters Vol. 20, No. 3, 209–212, 2013.
Acknowledgements
This work was supported in part by the National Key Research and Development Plan (Grant No. 2016YFC0801005), the National Natural Science Foundation of China (Grant Nos. 61772513 and 61402463), and the Open Foundation Project of Robot Technology Used for Special Environment Key Laboratory of Sichuan Province in China (Grant No. 16kftk01).
Author information
Authors and Affiliations
Corresponding author
Additional information
This article is published with open access at Springerlink.com
Shiming Ge is an associate professor in the Institute of Information Engineering, Chinese Academy of Sciences. He received his B.S. and Ph.D. degrees from the University of Science and Technology of China. His research mainly focuses on computer vision, deep learning, and AI security. He is a senior member of the IEEE and a member of the ACM.
Xin Jin received his Ph.D. degree in computer science from Beihang University. Currently, he is an assistant professor in Beijing Electronic Science and Technology Institute, China. His research mainly focuses on visual computing and visual media security.
Qiting Ye received his B.S. degree in computer science from Peking University in 2015. He is now a master student in the Institute of Information Engineering, Chinese Academy of Sciences. His major research interests lie in computer vision and deep learning.
Zhao Luo is currently a master candidate in the Institute of Information Engineering, Chinese Academy of Sciences. He received his B.S. degree from the University of Electronic Science and Technology of China. His research interests are object tracking and deep learning.
Qiang Li is currently a professor in Southwest University of Science and Technology. He received his Ph.D. degree from the University of Science and Technology of China. His research mainly focuses on the Internet of things and intelligent information processing.
Rights and permissions
Open Access The articles published in this journal are distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
Other papers from this open access journal are available free of charge from http://www.springer.com/journal/41095. To submit a manuscript, please go to https://www.editorialmanager.com/cvmj.
About this article
Cite this article
Ge, S., Jin, X., Ye, Q. et al. Image editing by object-aware optimal boundary searching and mixed-domain composition. Comp. Visual Media 4, 71–82 (2018). https://doi.org/10.1007/s41095-017-0102-8
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s41095-017-0102-8