Advertisement

Computational Visual Media

, Volume 1, Issue 4, pp 321–330 | Cite as

Color retargeting: Interactive time-varying color image composition from time-lapse sequences

  • Shao-Ping LuEmail author
  • Guillaume Dauphin
  • Gauthier Lafruit
  • Adrian Munteanu
Open Access
Research Article

Abstract

In this paper, we present an interactive static image composition approach, namely color retargeting, to flexibly represent time-varying color editing effect based on time-lapse video sequences. Instead of performing precise image matting or blending techniques, our approach treats the color composition as a pixel-level resampling problem. In order to both satisfy the user’s editing requirements and avoid visual artifacts, we construct a globally optimized interpolation field. This field defines from which input video frames the output pixels should be resampled. Our proposed resampling solution ensures that (i) the global color transition in the output image is as smooth as possible, (ii) the desired colors/objects specified by the user from different video frames are well preserved, and (iii) additional local color transition directions in the image space assigned by the user are also satisfied. Various examples have been shown to demonstrate that our efficient solution enables the user to easily create time-varying color image composition results.

Keywords

time-lapse sequence image composition color retargeting time-varying transition optimized interpolation field 

Supplementary material

41095_2015_31_MOESM1_ESM.mp4 (69.9 mb)
Supplementary material, approximately 71563 KB.

References

  1. [1]
    Wilkes, S. Day to night. 2015. Available at http://wwwstephenwilkescom/fine-art.Google Scholar
  2. [2]
    Setlur, V.; Takagi, S.; Raskar, R.; Gleicher, M.; Gooch, B. Automatic image retargeting. In: Proceedings of the 4th International Conference on Mobile and Ubiquitous Multimedia, 59–68, 2005.Google Scholar
  3. [3]
    Rubinstein, M.; Shamir, A.; Avidan, S. Improved seam carving for video retargeting. ACM Transactions on Graphics Vol. 27, No. 3, Article No. 16, 2008.CrossRefGoogle Scholar
  4. [4]
    Pérez, P.; Gangnet, M.; Blake, A. Poisson image editing. ACM Transactions on Graphics Vol. 22, No. 3, 313–318, 2003.CrossRefGoogle Scholar
  5. [5]
    Agarwala, A.; Dontcheva, M.; Agrawala, M.; Drucker, S.; Colburn, A.; Curless, B.; Salesin, D.; Cohen, M. Interactive digital photomontage. ACM Transactions on Graphics Vol. 23, No. 3, 294–302, 2004.CrossRefGoogle Scholar
  6. [6]
    Boykov, Y.; Veksler, O.; Zabih, R. Fast approximate energy minimization via graph cuts. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 23, No. 11, 1222–1239, 2001.CrossRefGoogle Scholar
  7. [7]
    Fattal, R.; Lischinski, D.; Werman, M. Gradient domain high dynamic range compression. ACM Transactions on Graphics Vol. 21, No. 3, 249–256, 2002.CrossRefGoogle Scholar
  8. [8]
    Pritch, Y.; Kav-Venaki, E.; Peleg, S. Shift-map image editing. In: Proceedings of IEEE 12th International Conference on Computer Vision, 151–158, 2009.Google Scholar
  9. [9]
    Farbman, Z.; Hoffer, G.; Lipman, Y.; Cohen-Or, D.; Lischinski, D. Coordinates for instant image cloning. ACM Transactions on Graphics Vol. 28, No. 3, Article No. 67, 2009.CrossRefGoogle Scholar
  10. [10]
    Darabi, S.; Shechtman, E.; Barnes, C.; Goldman, D. B.; Sen, P. Image melding: Combining inconsistent images using patch-based synthesis. ACM Transactions on Graphics Vol. 31, No. 4, Article No. 82, 2012.CrossRefGoogle Scholar
  11. [11]
    Avidan, S.; Shamir, A. Seam carving for content-aware image resizing. ACM Transactions on Graphics Vol. 26, No. 3, Article No. 10, 2007.CrossRefGoogle Scholar
  12. [12]
    Lin, S.-S.; Yeh, I.-C.; Lin, C.-H.; Lee, T.-Y. Patchbased image warping for content-aware retargeting. IEEE Transactions on Multimedia Vol. 15, No. 2, 359–368, 2013.Google Scholar
  13. [13]
    Zhu, Z.; Martin, R. R.; Hu, S.-M. Panorama completion for street views. Computational Visual Media Vol. 1, No. 1, 49–57, 2015.CrossRefGoogle Scholar
  14. [14]
    Lu, S.-P.; Hanca, J.; Munteanu, A.; Schelkens, P. Depth-based view synthesis using pixel-level image inpainting. In: Proceedings of the 18th International Conference on Digital Signal Processing, 1–6, 2013.Google Scholar
  15. [15]
    Levin, A.; Lischinski, D.; Weiss, Y. Colorization using optimization. ACM Transactions on Graphics Vol. 23, No. 3, 689–694, 2004.CrossRefGoogle Scholar
  16. [16]
    Lischinski, D.; Farbman, Z.; Uyttendaele, M.; Szeliski, R. Interactive local adjustment of tonal values. ACM Transactions on Graphics Vol. 25, No. 3, 646–653, 2006.CrossRefGoogle Scholar
  17. [17]
    An, X.; Pellacini, F. AppProp: All-pairs appearancespace edit propagation. ACM Transactions on Graphics Vol. 27, No. 3, Article No. 40, 2008.CrossRefGoogle Scholar
  18. [18]
    Xu, K.; Li, Y.; Ju, T.; Hu, S.-M.; Liu, T.-Q. Efficient affinity-based edit propagation using K-D tree. ACM Transactions on Graphics Vol. 28, No. 5, Article No. 118, 2009.Google Scholar
  19. [19]
    Farbman, Z.; Fattal, R.; Lischinski, D. Diffusion maps for edge-aware image editing. ACM Transactions on Graphics Vol. 29, No. 6, Article No. 145, 2010.CrossRefGoogle Scholar
  20. [20]
    Chen, X.; Zou, D.; Zhao, Q.; Tan, P. Manifold preserving edit propagation. ACM Transactions on Graphics Vol. 31, No. 6, Article No. 132, 2012.Google Scholar
  21. [21]
    Xu, L.; Yan, Q.; Jia, J. A sparse control model for image and video editing. ACM Transactions on Graphics Vol. 32, No. 6, Article No. 197, 2013.Google Scholar
  22. [22]
    Wong, B.-Y.; Shih, K.-T.; Liang, C.-K.; Chen, H. H. Single image realism assessment and recoloring by color compatibility. IEEE Transactions on Multimedia Vol. 14, No. 3, 760–769, 2012.Google Scholar
  23. [23]
    Ceulemans, B.; Lu, S.-P.; Schelkens, P.; Munteanu, A. Globally optimized multiview video color correction using dense spatio-temporal matching. In: Proceedings of 3DTV-Conference: The True Vision—Capture, Transmission and Display of 3D Video, 1–4, 2015.Google Scholar
  24. [24]
    Sunkavalli, K.; Matusik, W.; Pfister, H.; Rusinkiewicz, S. Factored time-lapse video. ACM Transactions on Graphics Vol. 26, No. 3, Article No. 101, 2007.CrossRefGoogle Scholar
  25. [25]
    Bennett, E. P.; McMillan, L. Computational timelapse video. ACM Transactions on Graphics Vol. 26, No. 3, Article No. 102, 2007.CrossRefGoogle Scholar
  26. [26]
    Correa, C. D.; Ma, K.-L. Dynamic video narratives. ACM Transactions on Graphics Vol. 29, No. 4, Article No. 88, 2010.CrossRefGoogle Scholar
  27. [27]
    Rav-Acha, A.; Pritch, Y.; Lischinski, D.; Peleg, S. Dynamosaicing: Mosaicing of dynamic scenes. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 29, No. 10, 1789–1801, 2007.CrossRefGoogle Scholar
  28. [28]
    Lu, S.-P.; Zhang, S.-H.; Wei, J.; Hu, S.-M.; Martin, R. R. Timeline editing of objects in video. IEEE Transactions on Visualization and Computer Graphics Vol. 19, No. 7, 1218–1227, 2013.CrossRefGoogle Scholar
  29. [29]
    Shih, Y.; Paris, S.; Durand, F.; Freeman, W. T. Datadriven hallucination of different times of day from a single outdoor photo. ACM Transactions on Graphics Vol. 32, No. 6, Article No. 200, 2013.Google Scholar
  30. [30]
    Estrada, F. J. Time-lapse image fusion. In: Proceedings of the 12th International Conference on Computer Vision, Vol. 2, 441–450, 2012.Google Scholar
  31. [31]
    Tan, J.; Dvorožňák, M.; Sýkora, D.; Gingold, Y. Decomposing time-lapse paintings into layers. ACM Transactions on Graphics Vol. 34, No. 4, Article No. 61, 2015.CrossRefGoogle Scholar
  32. [32]
    Martin-Brualla, R.; Gallup, D.; Seitz, S. M. Timelapse mining from internet photos. ACM Transactions on Graphics Vol. 34, No. 4, Article No. 62, 2015.Google Scholar
  33. [33]
    HaCohen, Y.; Shechtman, E.; Goldman, D. B.; Lischinski, D. Optimizing color consistency in photo collections. ACM Transactions on Graphics Vol. 32, No. 4, Article No. 38, 2013.CrossRefGoogle Scholar
  34. [34]
    Lu, S.-P.; Ceulemans, B.; Munteanu, A.; Schelkens, P. Spatiotemporally consistent color and structure optimization for multiview video color correction. IEEE Transactions on Multimedia Vol. 17, No. 5, 577–590, 2015.CrossRefGoogle Scholar
  35. [35]
    Eigen: A C++ linear algebra library. 2015. Available at http://eigentuxfamilyorg/index. php?title=Main Page.Google Scholar

Copyright information

© The Author(s) 2015

Authors and Affiliations

  • Shao-Ping Lu
    • 1
    Email author
  • Guillaume Dauphin
    • 2
  • Gauthier Lafruit
    • 2
  • Adrian Munteanu
    • 1
  1. 1.Department of Electronics and Informatics (ETRO)Vrije Universiteit Brussel (VUB) and iMindsBrusselsBelgium
  2. 2.Department of LISA, unit ImageUniversité Libre de Bruxelles (ULB)BrusselsBelgium

Personalised recommendations