Advertisement

Journal of Computer Science and Technology

, Volume 31, Issue 3, pp 463–478 | Cite as

View-Aware Image Object Compositing and Synthesis from Multiple Sources

  • Xiang Chen
  • Wei-Wei Xu
  • Sai-Kit Yeung
  • Kun ZhouEmail author
Regular Paper

Abstract

Image compositing is widely used to combine visual elements from separate source images into a single image. Although recent image compositing techniques are capable of achieving smooth blending of the visual elements from different sources, most of them implicitly assume the source images are taken in the same viewpoint. In this paper, we present an approach to compositing novel image objects from multiple source images which have different viewpoints. Our key idea is to construct 3D proxies for meaningful components of the source image objects, and use these 3D component proxies to warp and seamlessly merge components together in the same viewpoint. To realize this idea, we introduce a coordinate-frame based single-view camera calibration algorithm to handle general types of image objects, a structure-aware cuboid optimization algorithm to get the cuboid proxies for image object components with correct structure relationship, and finally a 3D-proxy transformation guided image warping algorithm to stitch object components. We further describe a novel application based on this compositing approach to automatically synthesize a large number of image objects from a set of exemplars. Experimental results show that our compositing approach can be applied to a variety of image objects, such as chairs, cups, lamps, and robots, and the synthesis application can create novel image objects with significant shape and style variations from a small set of exemplars.

Keywords

image cloning 3D proxy probabilistic modeling data-driven method 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. [1]
    Perez P, Gangnet M, Blake A. Poisson image editing. ACM Transactions on Graphics, 2003, 22(3): 313–318.CrossRefGoogle Scholar
  2. [2]
    Jia J, Sun J, Tang C K, Shum H Y. Drag-and-drop pasting. ACM Transactions on Graphics, 2006, 25(3): 631–637.CrossRefGoogle Scholar
  3. [3]
    Farbman Z, Hoffer G, Lipman Y, Cohen-Or D, Lischinski D. Coordinates for instant image cloning. ACM Transactions on Graphics, 2009, 28(3): Article No. 67.Google Scholar
  4. [4]
    Tao MW, Johnson M K, Paris S. Error-tolerant image compositing. In Proc. the 11th European Conference on Computer Vision, Sept. 2010, pp.31-44.Google Scholar
  5. [5]
    Sunkavalli K, Johnson M K, Matusik W, Pfister H. Multiscale image harmonization. ACM Transactions on Graphics, 2010, 29(4): Article No. 125.Google Scholar
  6. [6]
    Agarwala A, Dontcheva M, Agrawala M, Drucker S, Colburn A, Curless B, Salesin D, Cohen M. Interactive digital photomontage. ACM Transactions on Graphics, 2004, 23(3): 294–302.CrossRefGoogle Scholar
  7. [7]
    Rother C, Kumar S, Kolmogorov V, Blake A. Digital tapestry [automatic image synthesis]. In Proc. IEEE CVPR, June 2005, pp.589-596.Google Scholar
  8. [8]
    Rother C, Bordeaux L, Hamadi Y, Blake A. AutoCollage. ACM Transactions on Graphics, 2006, 25(3): 847–852.CrossRefGoogle Scholar
  9. [9]
    Wang J, Quan L, Sun J, Tang X, Shum H Y. Picture collage. In Proc. IEEE CVPR, June 2006, pp.347-354.Google Scholar
  10. [10]
    Chen T, Cheng M M, Tan P, Shamir A, Hu S M. Sketch2Photo: Internet image montage. ACM Transactions on Graphics, 2009, 28(5): 124:1–124:10.Google Scholar
  11. [11]
    Eitz M, Richter R, Hildebrand K, Boubekeur T, Alexa M. Photosketcher: Interactive sketch-based image synthesis. IEEE Computer Graphics and Applications, 2011, 31(6): 56–66.CrossRefGoogle Scholar
  12. [12]
    [12] Kalogerakis E, Chaudhuri S, Koller D, Koltun V. A probabilistic model for component-based shape synthesis. ACM Trans. Graph., 2012, 31(4): 55:1–55:11.CrossRefGoogle Scholar
  13. [13]
    Xu K, Zhang H, Cohen-Or D, Chen B. Fit and diverse: Set evolution for inspiring 3D shape galleries. ACM Trans. Graph., 2012, 31(4): 57:1–57:10.CrossRefGoogle Scholar
  14. [14]
    Burt P J, Adelson E H. A multiresolution spline with application to image mosaics. ACM Trans. Graph., 1983, 2(4): 217–236.CrossRefGoogle Scholar
  15. [15]
    Ogden J M, Adelson E H, Bergen J R, Burt P J. Pyramidbased computer graphics. RCA Engineer, 1985, 30(5): 4–15.Google Scholar
  16. [16]
    Porter T, Duff T. Compositing digital images. ACM SIGGRAPH Comput. Graph., 1984, 18(3): 253–259.CrossRefGoogle Scholar
  17. [17]
    Xue S, Agarwala A, Dorsey J, Rushmeier H. Understanding and improving the realism of image composites. ACM Transactions on Graphics, 2012, 31(4): Article No. 84.Google Scholar
  18. [18]
    Diakopoulos N, Essa I, Jain R. Content based image synthesis. In Proc. the 3rd CIVR, July 2004, pp.299-307.Google Scholar
  19. [19]
    Johnson M, Brostow G J, Shotton J et al. Semantic photo synthesis. Computer Graphics Forum, 2006, 25(3): 407–413.CrossRefGoogle Scholar
  20. [20]
    Lalonde J F, Hoiem D, Efros A A, Rother C, Winn J, Criminisi A. Photo clip art. ACM Transactions on Graphics, 2007, 26(3): Article No. 3.Google Scholar
  21. [21]
    Hall P, Cai H, Wu Q, Corradi T. Cross-depiction problem: Recognition and synthesis of photographs and artwork. Computational Visual Media, 2015, 1(2): 91–103.CrossRefGoogle Scholar
  22. [22]
    Huang H, Zhang L, Zhang H C. Arcimboldo-like collage using internet images. ACM Transactions on Graphics, 2011, 30(6): Article No. 155.Google Scholar
  23. [23]
    Yu Z, Lu L, Guo Y, Fan R, Liu M, Wang W. Content-aware photo collage using circle packing. IEEE Transactions on Visualization and Computer Graphics, 2014, 20(2): 182-195.CrossRefGoogle Scholar
  24. [24]
    Risser E, Han C, Dahyot R, Grinspun E. Synthesizing structured image hybrids. ACM Transactions on Graphics, 2010, 29(4): Article No. 85.Google Scholar
  25. [25]
    Carroll R, Agarwala A, Agrawala M. Image warps for artistic perspective manipulation. ACM Transactions on Graphics, 2010, 29(4): Article No. 127.Google Scholar
  26. [26]
    Zheng Y, Chen X, Cheng M M, Zhou K, Hu S M, Mitra N J. Interactive images: Cuboid proxies for smart image manipulation. ACM Trans. Graph., 2012, 31(4): 99:1–99:11.Google Scholar
  27. [27]
    Chen T, Zhu Z, Shamir A, Hu S M, Cohen-Or D. 3-sweep: Extracting editable objects from a single photo. ACM Transactions on Graphics, 2013, 32(6): Article No. 195.Google Scholar
  28. [28]
    Miao Y, Hu F, Zhang X, Chen J, Pajarola R. SymmSketch: Creating symmetric 3D free-form shapes from 2D sketches. Computational Visual Media, 2015, 1(1): 3–16.CrossRefGoogle Scholar
  29. [29]
    Funkhouser T, Kazhdan M, Shilane P, Min P, KieferW, Tal A, Rusinkiewicz S, Dobkin D. Modeling by example. ACM Trans. Graph., 2004, 23(3): 652–663.CrossRefGoogle Scholar
  30. [30]
    Shin H, Igarashi T. Magic canvas: Interactive design of a 3-D scene prototype from freehand sketches. In Proc. Graphics Interface, May 2007, pp.63-70.Google Scholar
  31. [31]
    Lee J, Funkhouser T. Sketch-based search and composition of 3D models. In Proc. the 5th SBM, June 2008, pp.97-104.Google Scholar
  32. [32]
    Xu K, Chen K, Fu H, Sun W L, Hu S M. Sketch2Scene: Sketch-based co-retrieval and co-placement of 3D models. ACM Transactions on Graphics, 2013, 32(4): Article No. 123.Google Scholar
  33. [33]
    Kreavoy V, Julius D, Sheffer A. Model composition from interchangeable components. In Proc. the 15th PG, Oct. 2007, pp.129-138.Google Scholar
  34. [34]
    Chaudhuri S, Koltun V. Data-driven suggestions for creativity support in 3D modeling. ACM Trans. Graph., 2010, 29(6): 183:1–183:10.CrossRefGoogle Scholar
  35. [35]
    Chaudhuri S, Kalogerakis E, Guibas L, Koltun V. Probabilistic reasoning for assembly-based 3D modeling. ACM Trans. Graph., 2011, 30(4): 35:1–35:10.CrossRefGoogle Scholar
  36. [36]
    Li Y, Sun J, Tang C K, Shum H Y. Lazy snapping. ACM Transactions on Graphics, 2004, 23(3): 303–308.CrossRefGoogle Scholar
  37. [37]
    Russell B C, Torralba A, Murphy K P, Freeman W T. LabelMe: A database and web-based tool for image annotation. International Journal of Computer Vision, 2008, 77(1/2/3): 157–173.Google Scholar
  38. [38]
    Barnes C, Shechtman E, Finkelstein A, Goldman D B. PatchMatch: A randomized correspondence algorithm for structural image editing. ACM Transactions on Graphics, 2009, 28(3): 24:1–24:11.CrossRefGoogle Scholar
  39. [39]
    Criminisi A, Reid I, Zisserman A. Single view metrology. International Journal of Computer Vision, 2000, 40(2): 123-148.CrossRefzbMATHGoogle Scholar
  40. [40]
    Sinha S N, Steedly D, Szeliski R, Agrawala M, Pollefeys M. Interactive 3D architectural modeling from unordered photo collections. ACM Transactions on Graphics, 2008, 27(5): 159:1–159:10.CrossRefGoogle Scholar
  41. [41]
    Wilczkowiak M, Sturm P, Boyer E. Using geometric constraints through parallelepipeds for calibration and 3D modeling. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2005, 27(2): 194–207.CrossRefGoogle Scholar
  42. [42]
    Jiang N, Tan P, Cheong L F. Symmetric architecture modeling with a single image. ACM Transactions on Graphics, 2009, 28(5): 113:1–113:8.CrossRefGoogle Scholar
  43. [43]
    Shen C H, Fu H, Chen K, Hu S M. Structure recovery by part assembly. ACM Transactions on Graphics, 2012, 31(6): Article No. 180.Google Scholar
  44. [44]
    O’Donovan P, Agarwala A, Hertzmann A. Color compatibility from large datasets. ACM Transactions on Graphics, 2011, 30(4): Article No. 63.Google Scholar
  45. [45]
    Yu L F, Yeung S K, Terzopoulos D, Chan T F. DressUp!: Outfit synthesis through automatic optimization. ACM Transactions on Graphics, 2012, 31(6): 134:1–134:14.CrossRefGoogle Scholar
  46. [46]
    Xu K, Zheng H, Zhang H, Cohen-Or D, Liu L, Xiong Y. Photo-inspired model-driven 3D object modeling. ACM Trans. Graph., 2011, 30(4): 80:1–80:10.Google Scholar
  47. [47]
    Cootes T F, Taylor C J, Cooper D H, Graham J et al. Active shape models — Their training and application. Computer Vision and Image Understanding, 1995, 61(1): 38–59.CrossRefGoogle Scholar
  48. [48]
    Schwarz G. Estimating the dimension of a model. The Annals of Statistics, 1978, 6(2): 461–464.MathSciNetCrossRefzbMATHGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2016

Authors and Affiliations

  • Xiang Chen
    • 1
  • Wei-Wei Xu
    • 1
  • Sai-Kit Yeung
    • 2
  • Kun Zhou
    • 1
    Email author
  1. 1.State Key Laboratory of Computer Aided Design and Computer GraphicsZhejiang UniversityHangzhouChina
  2. 2.Vision, Graphics and Computational Design GroupSingapore University of Technology and DesignSingaporeSingapore

Personalised recommendations