Skip to main content
Download PDF

Image editing by object-aware optimal boundary searching and mixed-domain composition

Computational Visual Media volume 4, pages 71–82 (2018)Cite this article

Abstract

When combining very different images which often contain complex objects and backgrounds, producing consistent compositions is a challenging problem requiring seamless image editing. In this paper, we propose a general approach, called object-aware image editing, to obtain consistency in structure, color, and texture in a unified way. Our approach improves upon previous gradient-domain composition in three ways. Firstly, we introduce an iterative optimization algorithm to minimize mismatches on the boundaries when the target region contains multiple objects of interest. Secondly, we propose a mixed-domain consistency metric for measuring gradients and colors, and formulate composition as a unified minimization problem that can be solved with a sparse linear system. In particular, we encode texture consistency using a patch-based approach without searching and matching. Thirdly, we adopt an object-aware approach to separately manipulate the guidance gradient fields for objects of interest and backgrounds of interest, which facilitates a variety of seamless image editing applications. Our unified method outperforms previous state-of-the-art methods in preserving global texture consistency in addition to local structure continuity.

References

  1. Philip, S.; Summa, B.; Tierny, J.; Bremer, P. T.; Pascucci, V. Distributed seams for gigapixel panoramas. IEEE Transactions on Visualization and Computer Graphics Vol. 21, No. 3, 350–362, 2015.

    Article  Google Scholar 

  2. Agarwala, A.; Dontcheva, M.; Agrawala, M.; Drucker, S.; Colburn, A.; Curless, B.; Salesin, D.; Cohen, M. Interactive digital photomontage. ACM Transactions on Graphics Vol. 23, No. 3, 294–302, 2004.

    Article  Google Scholar 

  3. Cheng, M.-M.; Zhang, F.-L.; Mitra, N. J.; Huang, X.; Hu, S.-M. RepFinder: Finding approximately repeated scene elements for image editing. ACM Transactions on Graphics Vol. 29, No. 4, Article No. 83, 2010.

    Google Scholar 

  4. Barnes, C.; Zhang, F.-L.; Lou, L.; Wu, X.; Hu, S.-M. PatchTable: Efficient patch queries for large datasets and applications. ACM Transactions on Graphics Vol. 34, No. 4, Article No. 97, 2015.

    Google Scholar 

  5. Li, J.; Tian, Y.; Huang, T. Visual saliency with statistical priors. International Journal of Computer Vision Vol. 107, No. 3, 239–253, 2014.

    MathSciNet  Article  MATH  Google Scholar 

  6. Li, J.; Duan, L. Y.; Chen, X.; Huang, T.; Tian, Y. Finding the secret of image saliency in the frequency domain. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 37, No. 12, 2428–2440, 2015.

    Article  Google Scholar 

  7. Efros, A. A.; Freeman, W. T. Image quilting for texture synthesis and transfer. In: Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, 341–346, 2001.

    Google Scholar 

  8. Kwatra, V.; Schödl, A.; Essa, I.; Turk, G.; Bobick, A. Graphcut textures: Image and video synthesis using graph cuts. ACM Transactions on Graphics Vol. 22, No. 3, 277–286, 2003.

    Article  Google Scholar 

  9. Darabi, S.; Shechtman, E.; Barnes, C.; Goldman, D. B.; Sen, P. Image melding: Combining inconsistent images using patch-based synthesis. ACM Transactions on Graphics Vol. 31, No. 4, Article No. 82, 2012.

    Google Scholar 

  10. Tao, M. W.; Johnson, M. K.; Paris, S. Error-tolerant image compositing. International Journal of Computer Vision Vol. 103, No. 2, 178–189, 2013.

    Article  MATH  Google Scholar 

  11. Pérez, P.; Gangnet, M.; Blake, A. Poisson image editing. ACM Transactions on Graphics Vol. 22, No. 3, 313–318, 2003.

    Article  Google Scholar 

  12. Zomet, A.; Levin, A.; Peleg, S.; Weiss, Y. Seamless image stitching by minimizing false edges. IEEE Transactions on Image Processing Vol. 15, No. 4, 969–977, 2006.

    Article  Google Scholar 

  13. Jia, J.; Sun, J.; Tang, C.-K.; Shum, H.-Y. Drag-anddrop pasting. ACM Transactions on Graphics Vol. 25, No. 3, 631–637, 2006.

    Article  Google Scholar 

  14. Farbman, Z.; Hoffer, G.; Lipman, Y.; Cohen-Or, D.; Lischinski, D. Coordinates for instant image cloning. ACM Transactions on Graphics Vol. 28, No. 3, Article No. 67, 2009.

    Google Scholar 

  15. Bhat, P.; Zitnick, C. L.; Cohen, M.; Curless, B. GradientShop: A gradient-domain optimization framework for image and video filtering. ACM Transactions on Graphics Vol. 29, No. 2, Article No. 10, 2010.

    Google Scholar 

  16. Li, X. Y.; Gu, Y.; Hu, S.-M.; Martin, R. R. Mixed-domain edge-aware image manipulation. IEEE Transactions on Image Processing Vol. 22, No. 5, 1915–1925, 2013.

    MathSciNet  Article  MATH  Google Scholar 

  17. Sadek, R.; Facciolo, G.; Arias, P.; Caselles, V. A variational model for gradient-based video editing. International Journal of Computer Vision Vol. 103, No. 1, 127–162, 2013.

    MathSciNet  Article  MATH  Google Scholar 

  18. Bie, X.; Wang, W.; Sun, H.; Huang, H.; Zhang, M. Intent-aware image cloning. The Visual Computer Vol. 29, Nos. 6–8, 599–608, 2013.

    Article  Google Scholar 

  19. Hua, M.; Bie, X.; Zhang, M.; Wang, W. Edge-aware gradient domain optimization framework for image filtering by local propagation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2838–2845, 2014.

    Google Scholar 

  20. Zhang, Y.; Ling, J.; Zhang, X.; Xie, H. Image copy-and-paste with optimized gradient. The Visual Computer Vol. 30, No. 10, 1169–1178, 2014.

    Article  Google Scholar 

  21. Ma, L.-Q.; Xu, K. Efficient manifold preserving edit propagation with adaptive neighborhood size. Computers & Graphics Vol. 38, 167–173, 2014.

    Article  Google Scholar 

  22. Luo, S. J.; Sun, Y. T.; Shen, I. C.; Chen, B. Y.; Chuang, Y. Y. Geometrically consistent stereoscopic image editing using patch-based synthesis. IEEE Transactions on Visualization and Computer Graphics Vol. 21, No. 1, 56–67, 2015.

    Article  Google Scholar 

  23. Chen, T.; Cheng, M.-M.; Tan, P.; Shamir, A.; Hu, S.-M. Sketch2Photo: Internet image montage. ACM Transactions on Graphics Vol. 28, No. 5, Article No. 124, 2009.

    Google Scholar 

  24. Zhang, F. L.; Wang, J.; Shechtman, E.; Zhou, Z. Y.; Shi, J. X.; Hu, S. M. PlenoPatch: Patch-based plenoptic image manipulation. IEEE Transactions on Visualization and Computer Graphics Vol. 23, No. 5, 1561–1573, 2016.

    Article  Google Scholar 

  25. Lee, J. H.; Choi, I.; Kim, M. H. Laplacian patchbased image synthesis. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2727–2735, 2016.

    Google Scholar 

  26. Mortensen, E. N.; Barrett, W. A. Intelligent scissors for image composition. In: Proceedings of the 22nd Annual Conference on Computer Graphics and Interactive Techniques, 191–198, 1995.

    Google Scholar 

  27. Sethian, J. Level Set Methods and Fast Marching Methods: Evolving Interfaces in Computational Geometry, Fluid Mechanics, Computer Vision, and Materials Sciences. Cambridge University Press, 1999.

    MATH  Google Scholar 

  28. Rother, C.; Kolmogorov, V.; Blake, A. “GrabCut”: Interactive foreground extraction using iterated graph cuts. ACM Transactions on Graphics Vol. 23, No. 3, 309–314, 2004.

    Article  Google Scholar 

  29. Krishnan, D.; Szeliski, R. Multigrid and multilevel preconditioners for computational photography. ACM Transactions on Graphics Vol. 30, No. 6, Article No. 177, 2011.

    Google Scholar 

  30. Mittal, A.; Soundararajan, R.; Bovik, A. C. Making a “completely blind” image quality analyzer. IEEE Signal Processing Letters Vol. 20, No. 3, 209–212, 2013.

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported in part by the National Key Research and Development Plan (Grant No. 2016YFC0801005), the National Natural Science Foundation of China (Grant Nos. 61772513 and 61402463), and the Open Foundation Project of Robot Technology Used for Special Environment Key Laboratory of Sichuan Province in China (Grant No. 16kftk01).

Author information

Authors and Affiliations

  1. Institute of Information Engineering, Chinese Academy of Sciences, Beijing, 100095, China

    Shiming Ge, Qiting Ye & Zhao Luo

  2. Beijing Electronic Science and Technology Institute, Beijing, 100070, China

    Xin Jin

  3. School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China

    Qiting Ye & Zhao Luo

  4. School of Information Engineering, Southwest University of Science and Technology, Mianyang, 621010, China

    Qiang Li

Authors
  1. Shiming Ge
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xin Jin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Qiting Ye
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zhao Luo
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Qiang Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shiming Ge.

Additional information

This article is published with open access at Springerlink.com

Shiming Ge is an associate professor in the Institute of Information Engineering, Chinese Academy of Sciences. He received his B.S. and Ph.D. degrees from the University of Science and Technology of China. His research mainly focuses on computer vision, deep learning, and AI security. He is a senior member of the IEEE and a member of the ACM.

Xin Jin received his Ph.D. degree in computer science from Beihang University. Currently, he is an assistant professor in Beijing Electronic Science and Technology Institute, China. His research mainly focuses on visual computing and visual media security.

Qiting Ye received his B.S. degree in computer science from Peking University in 2015. He is now a master student in the Institute of Information Engineering, Chinese Academy of Sciences. His major research interests lie in computer vision and deep learning.

Zhao Luo is currently a master candidate in the Institute of Information Engineering, Chinese Academy of Sciences. He received his B.S. degree from the University of Electronic Science and Technology of China. His research interests are object tracking and deep learning.

Qiang Li is currently a professor in Southwest University of Science and Technology. He received his Ph.D. degree from the University of Science and Technology of China. His research mainly focuses on the Internet of things and intelligent information processing.

Rights and permissions

Open Access The articles published in this journal are distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Other papers from this open access journal are available free of charge from http://www.springer.com/journal/41095. To submit a manuscript, please go to https://www.editorialmanager.com/cvmj.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Ge, S., Jin, X., Ye, Q. et al. Image editing by object-aware optimal boundary searching and mixed-domain composition. Comp. Visual Media 4, 71–82 (2018). https://doi.org/10.1007/s41095-017-0102-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s41095-017-0102-8

Keywords

  • seamless image editing
  • patch-based synthesis
  • image composition
  • mixed-domain
  • gradient-domain composition