Download PDF

Computational Visual Media

, Volume 4, Issue 1, pp 71–82 | Cite as

Image editing by object-aware optimal boundary searching and mixed-domain composition

Open Access
Research Article
First Online:
Received:
Accepted:
  • 67 Downloads

Abstract

When combining very different images which often contain complex objects and backgrounds, producing consistent compositions is a challenging problem requiring seamless image editing. In this paper, we propose a general approach, called object-aware image editing, to obtain consistency in structure, color, and texture in a unified way. Our approach improves upon previous gradient-domain composition in three ways. Firstly, we introduce an iterative optimization algorithm to minimize mismatches on the boundaries when the target region contains multiple objects of interest. Secondly, we propose a mixed-domain consistency metric for measuring gradients and colors, and formulate composition as a unified minimization problem that can be solved with a sparse linear system. In particular, we encode texture consistency using a patch-based approach without searching and matching. Thirdly, we adopt an object-aware approach to separately manipulate the guidance gradient fields for objects of interest and backgrounds of interest, which facilitates a variety of seamless image editing applications. Our unified method outperforms previous state-of-the-art methods in preserving global texture consistency in addition to local structure continuity.

Keywords

seamless image editing patch-based synthesis image composition mixed-domain gradient-domain composition 
Download to read the full article text

Notes

Acknowledgements

This work was supported in part by the National Key Research and Development Plan (Grant No. 2016YFC0801005), the National Natural Science Foundation of China (Grant Nos. 61772513 and 61402463), and the Open Foundation Project of Robot Technology Used for Special Environment Key Laboratory of Sichuan Province in China (Grant No. 16kftk01).

References

  1. [1]
    Philip, S.; Summa, B.; Tierny, J.; Bremer, P. T.; Pascucci, V. Distributed seams for gigapixel panoramas. IEEE Transactions on Visualization and Computer Graphics Vol. 21, No. 3, 350–362, 2015.CrossRefGoogle Scholar
  2. [2]
    Agarwala, A.; Dontcheva, M.; Agrawala, M.; Drucker, S.; Colburn, A.; Curless, B.; Salesin, D.; Cohen, M. Interactive digital photomontage. ACM Transactions on Graphics Vol. 23, No. 3, 294–302, 2004.CrossRefGoogle Scholar
  3. [3]
    Cheng, M.-M.; Zhang, F.-L.; Mitra, N. J.; Huang, X.; Hu, S.-M. RepFinder: Finding approximately repeated scene elements for image editing. ACM Transactions on Graphics Vol. 29, No. 4, Article No. 83, 2010.Google Scholar
  4. [4]
    Barnes, C.; Zhang, F.-L.; Lou, L.; Wu, X.; Hu, S.-M. PatchTable: Efficient patch queries for large datasets and applications. ACM Transactions on Graphics Vol. 34, No. 4, Article No. 97, 2015.Google Scholar
  5. [5]
    Li, J.; Tian, Y.; Huang, T. Visual saliency with statistical priors. International Journal of Computer Vision Vol. 107, No. 3, 239–253, 2014.MathSciNetCrossRefMATHGoogle Scholar
  6. [6]
    Li, J.; Duan, L. Y.; Chen, X.; Huang, T.; Tian, Y. Finding the secret of image saliency in the frequency domain. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 37, No. 12, 2428–2440, 2015.CrossRefGoogle Scholar
  7. [7]
    Efros, A. A.; Freeman, W. T. Image quilting for texture synthesis and transfer. In: Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, 341–346, 2001.Google Scholar
  8. [8]
    Kwatra, V.; Schödl, A.; Essa, I.; Turk, G.; Bobick, A. Graphcut textures: Image and video synthesis using graph cuts. ACM Transactions on Graphics Vol. 22, No. 3, 277–286, 2003.CrossRefGoogle Scholar
  9. [9]
    Darabi, S.; Shechtman, E.; Barnes, C.; Goldman, D. B.; Sen, P. Image melding: Combining inconsistent images using patch-based synthesis. ACM Transactions on Graphics Vol. 31, No. 4, Article No. 82, 2012.Google Scholar
  10. [10]
    Tao, M. W.; Johnson, M. K.; Paris, S. Error-tolerant image compositing. International Journal of Computer Vision Vol. 103, No. 2, 178–189, 2013.CrossRefMATHGoogle Scholar
  11. [11]
    Pérez, P.; Gangnet, M.; Blake, A. Poisson image editing. ACM Transactions on Graphics Vol. 22, No. 3, 313–318, 2003.CrossRefGoogle Scholar
  12. [12]
    Zomet, A.; Levin, A.; Peleg, S.; Weiss, Y. Seamless image stitching by minimizing false edges. IEEE Transactions on Image Processing Vol. 15, No. 4, 969–977, 2006.CrossRefGoogle Scholar
  13. [13]
    Jia, J.; Sun, J.; Tang, C.-K.; Shum, H.-Y. Drag-anddrop pasting. ACM Transactions on Graphics Vol. 25, No. 3, 631–637, 2006.CrossRefGoogle Scholar
  14. [14]
    Farbman, Z.; Hoffer, G.; Lipman, Y.; Cohen-Or, D.; Lischinski, D. Coordinates for instant image cloning. ACM Transactions on Graphics Vol. 28, No. 3, Article No. 67, 2009.Google Scholar
  15. [15]
    Bhat, P.; Zitnick, C. L.; Cohen, M.; Curless, B. GradientShop: A gradient-domain optimization framework for image and video filtering. ACM Transactions on Graphics Vol. 29, No. 2, Article No. 10, 2010.Google Scholar
  16. [16]
    Li, X. Y.; Gu, Y.; Hu, S.-M.; Martin, R. R. Mixed-domain edge-aware image manipulation. IEEE Transactions on Image Processing Vol. 22, No. 5, 1915–1925, 2013.MathSciNetCrossRefMATHGoogle Scholar
  17. [17]
    Sadek, R.; Facciolo, G.; Arias, P.; Caselles, V. A variational model for gradient-based video editing. International Journal of Computer Vision Vol. 103, No. 1, 127–162, 2013.MathSciNetCrossRefMATHGoogle Scholar
  18. [18]
    Bie, X.; Wang, W.; Sun, H.; Huang, H.; Zhang, M. Intent-aware image cloning. The Visual Computer Vol. 29, Nos. 6–8, 599–608, 2013.CrossRefGoogle Scholar
  19. [19]
    Hua, M.; Bie, X.; Zhang, M.; Wang, W. Edge-aware gradient domain optimization framework for image filtering by local propagation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2838–2845, 2014.Google Scholar
  20. [20]
    Zhang, Y.; Ling, J.; Zhang, X.; Xie, H. Image copy-and-paste with optimized gradient. The Visual Computer Vol. 30, No. 10, 1169–1178, 2014.CrossRefGoogle Scholar
  21. [21]
    Ma, L.-Q.; Xu, K. Efficient manifold preserving edit propagation with adaptive neighborhood size. Computers & Graphics Vol. 38, 167–173, 2014.CrossRefGoogle Scholar
  22. [22]
    Luo, S. J.; Sun, Y. T.; Shen, I. C.; Chen, B. Y.; Chuang, Y. Y. Geometrically consistent stereoscopic image editing using patch-based synthesis. IEEE Transactions on Visualization and Computer Graphics Vol. 21, No. 1, 56–67, 2015.CrossRefGoogle Scholar
  23. [23]
    Chen, T.; Cheng, M.-M.; Tan, P.; Shamir, A.; Hu, S.-M. Sketch2Photo: Internet image montage. ACM Transactions on Graphics Vol. 28, No. 5, Article No. 124, 2009.Google Scholar
  24. [24]
    Zhang, F. L.; Wang, J.; Shechtman, E.; Zhou, Z. Y.; Shi, J. X.; Hu, S. M. PlenoPatch: Patch-based plenoptic image manipulation. IEEE Transactions on Visualization and Computer Graphics Vol. 23, No. 5, 1561–1573, 2016.CrossRefGoogle Scholar
  25. [25]
    Lee, J. H.; Choi, I.; Kim, M. H. Laplacian patchbased image synthesis. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2727–2735, 2016.Google Scholar
  26. [26]
    Mortensen, E. N.; Barrett, W. A. Intelligent scissors for image composition. In: Proceedings of the 22nd Annual Conference on Computer Graphics and Interactive Techniques, 191–198, 1995.Google Scholar
  27. [27]
    Sethian, J. Level Set Methods and Fast Marching Methods: Evolving Interfaces in Computational Geometry, Fluid Mechanics, Computer Vision, and Materials Sciences. Cambridge University Press, 1999.MATHGoogle Scholar
  28. [28]
    Rother, C.; Kolmogorov, V.; Blake, A. “GrabCut”: Interactive foreground extraction using iterated graph cuts. ACM Transactions on Graphics Vol. 23, No. 3, 309–314, 2004.CrossRefGoogle Scholar
  29. [29]
    Krishnan, D.; Szeliski, R. Multigrid and multilevel preconditioners for computational photography. ACM Transactions on Graphics Vol. 30, No. 6, Article No. 177, 2011.Google Scholar
  30. [30]
    Mittal, A.; Soundararajan, R.; Bovik, A. C. Making a “completely blind” image quality analyzer. IEEE Signal Processing Letters Vol. 20, No. 3, 209–212, 2013.CrossRefGoogle Scholar

Copyright information

© The Author(s) 2017

Open Access The articles published in this journal are distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Other papers from this open access journal are available free of charge from http://www.springer.com/journal/41095. To submit a manuscript, please go to https://www.editorialmanager.com/cvmj.

Authors and Affiliations

  • Shiming Ge
    • 1
  • Xin Jin
    • 2
  • Qiting Ye
    • 1
    • 3
  • Zhao Luo
    • 1
    • 3
  • Qiang Li
    • 4
  1. 1.Institute of Information EngineeringChinese Academy of SciencesBeijingChina
  2. 2.Beijing Electronic Science and Technology InstituteBeijingChina
  3. 3.School of Cyber SecurityUniversity of Chinese Academy of SciencesBeijingChina
  4. 4.School of Information EngineeringSouthwest University of Science and TechnologyMianyangChina

Personalised recommendations