Abstract
In this paper, we present an interactive static image composition approach, namely color retargeting, to flexibly represent time-varying color editing effect based on time-lapse video sequences. Instead of performing precise image matting or blending techniques, our approach treats the color composition as a pixel-level resampling problem. In order to both satisfy the user’s editing requirements and avoid visual artifacts, we construct a globally optimized interpolation field. This field defines from which input video frames the output pixels should be resampled. Our proposed resampling solution ensures that (i) the global color transition in the output image is as smooth as possible, (ii) the desired colors/objects specified by the user from different video frames are well preserved, and (iii) additional local color transition directions in the image space assigned by the user are also satisfied. Various examples have been shown to demonstrate that our efficient solution enables the user to easily create time-varying color image composition results.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
References
Wilkes, S. Day to night. 2015. Available at http://wwwstephenwilkescom/fine-art.
Setlur, V.; Takagi, S.; Raskar, R.; Gleicher, M.; Gooch, B. Automatic image retargeting. In: Proceedings of the 4th International Conference on Mobile and Ubiquitous Multimedia, 59–68, 2005.
Rubinstein, M.; Shamir, A.; Avidan, S. Improved seam carving for video retargeting. ACM Transactions on Graphics Vol. 27, No. 3, Article No. 16, 2008.
Pérez, P.; Gangnet, M.; Blake, A. Poisson image editing. ACM Transactions on Graphics Vol. 22, No. 3, 313–318, 2003.
Agarwala, A.; Dontcheva, M.; Agrawala, M.; Drucker, S.; Colburn, A.; Curless, B.; Salesin, D.; Cohen, M. Interactive digital photomontage. ACM Transactions on Graphics Vol. 23, No. 3, 294–302, 2004.
Boykov, Y.; Veksler, O.; Zabih, R. Fast approximate energy minimization via graph cuts. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 23, No. 11, 1222–1239, 2001.
Fattal, R.; Lischinski, D.; Werman, M. Gradient domain high dynamic range compression. ACM Transactions on Graphics Vol. 21, No. 3, 249–256, 2002.
Pritch, Y.; Kav-Venaki, E.; Peleg, S. Shift-map image editing. In: Proceedings of IEEE 12th International Conference on Computer Vision, 151–158, 2009.
Farbman, Z.; Hoffer, G.; Lipman, Y.; Cohen-Or, D.; Lischinski, D. Coordinates for instant image cloning. ACM Transactions on Graphics Vol. 28, No. 3, Article No. 67, 2009.
Darabi, S.; Shechtman, E.; Barnes, C.; Goldman, D. B.; Sen, P. Image melding: Combining inconsistent images using patch-based synthesis. ACM Transactions on Graphics Vol. 31, No. 4, Article No. 82, 2012.
Avidan, S.; Shamir, A. Seam carving for content-aware image resizing. ACM Transactions on Graphics Vol. 26, No. 3, Article No. 10, 2007.
Lin, S.-S.; Yeh, I.-C.; Lin, C.-H.; Lee, T.-Y. Patchbased image warping for content-aware retargeting. IEEE Transactions on Multimedia Vol. 15, No. 2, 359–368, 2013.
Zhu, Z.; Martin, R. R.; Hu, S.-M. Panorama completion for street views. Computational Visual Media Vol. 1, No. 1, 49–57, 2015.
Lu, S.-P.; Hanca, J.; Munteanu, A.; Schelkens, P. Depth-based view synthesis using pixel-level image inpainting. In: Proceedings of the 18th International Conference on Digital Signal Processing, 1–6, 2013.
Levin, A.; Lischinski, D.; Weiss, Y. Colorization using optimization. ACM Transactions on Graphics Vol. 23, No. 3, 689–694, 2004.
Lischinski, D.; Farbman, Z.; Uyttendaele, M.; Szeliski, R. Interactive local adjustment of tonal values. ACM Transactions on Graphics Vol. 25, No. 3, 646–653, 2006.
An, X.; Pellacini, F. AppProp: All-pairs appearancespace edit propagation. ACM Transactions on Graphics Vol. 27, No. 3, Article No. 40, 2008.
Xu, K.; Li, Y.; Ju, T.; Hu, S.-M.; Liu, T.-Q. Efficient affinity-based edit propagation using K-D tree. ACM Transactions on Graphics Vol. 28, No. 5, Article No. 118, 2009.
Farbman, Z.; Fattal, R.; Lischinski, D. Diffusion maps for edge-aware image editing. ACM Transactions on Graphics Vol. 29, No. 6, Article No. 145, 2010.
Chen, X.; Zou, D.; Zhao, Q.; Tan, P. Manifold preserving edit propagation. ACM Transactions on Graphics Vol. 31, No. 6, Article No. 132, 2012.
Xu, L.; Yan, Q.; Jia, J. A sparse control model for image and video editing. ACM Transactions on Graphics Vol. 32, No. 6, Article No. 197, 2013.
Wong, B.-Y.; Shih, K.-T.; Liang, C.-K.; Chen, H. H. Single image realism assessment and recoloring by color compatibility. IEEE Transactions on Multimedia Vol. 14, No. 3, 760–769, 2012.
Ceulemans, B.; Lu, S.-P.; Schelkens, P.; Munteanu, A. Globally optimized multiview video color correction using dense spatio-temporal matching. In: Proceedings of 3DTV-Conference: The True Vision—Capture, Transmission and Display of 3D Video, 1–4, 2015.
Sunkavalli, K.; Matusik, W.; Pfister, H.; Rusinkiewicz, S. Factored time-lapse video. ACM Transactions on Graphics Vol. 26, No. 3, Article No. 101, 2007.
Bennett, E. P.; McMillan, L. Computational timelapse video. ACM Transactions on Graphics Vol. 26, No. 3, Article No. 102, 2007.
Correa, C. D.; Ma, K.-L. Dynamic video narratives. ACM Transactions on Graphics Vol. 29, No. 4, Article No. 88, 2010.
Rav-Acha, A.; Pritch, Y.; Lischinski, D.; Peleg, S. Dynamosaicing: Mosaicing of dynamic scenes. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 29, No. 10, 1789–1801, 2007.
Lu, S.-P.; Zhang, S.-H.; Wei, J.; Hu, S.-M.; Martin, R. R. Timeline editing of objects in video. IEEE Transactions on Visualization and Computer Graphics Vol. 19, No. 7, 1218–1227, 2013.
Shih, Y.; Paris, S.; Durand, F.; Freeman, W. T. Datadriven hallucination of different times of day from a single outdoor photo. ACM Transactions on Graphics Vol. 32, No. 6, Article No. 200, 2013.
Estrada, F. J. Time-lapse image fusion. In: Proceedings of the 12th International Conference on Computer Vision, Vol. 2, 441–450, 2012.
Tan, J.; Dvorožňák, M.; Sýkora, D.; Gingold, Y. Decomposing time-lapse paintings into layers. ACM Transactions on Graphics Vol. 34, No. 4, Article No. 61, 2015.
Martin-Brualla, R.; Gallup, D.; Seitz, S. M. Timelapse mining from internet photos. ACM Transactions on Graphics Vol. 34, No. 4, Article No. 62, 2015.
HaCohen, Y.; Shechtman, E.; Goldman, D. B.; Lischinski, D. Optimizing color consistency in photo collections. ACM Transactions on Graphics Vol. 32, No. 4, Article No. 38, 2013.
Lu, S.-P.; Ceulemans, B.; Munteanu, A.; Schelkens, P. Spatiotemporally consistent color and structure optimization for multiview video color correction. IEEE Transactions on Multimedia Vol. 17, No. 5, 577–590, 2015.
Eigen: A C++ linear algebra library. 2015. Available at http://eigentuxfamilyorg/index. php?title=Main Page.
Author information
Authors and Affiliations
Corresponding author
Additional information
Shao-Ping Lu is currently a senior researcher at Vrije Universiteit Brussel (VUB), Brussels, Belgium, where he was a post-doc researcher from Nov. 2012 to Dec. 2014. He received his Ph.D. degree in computer science from Tsinghua University, Beijing, China, in July 2012. His research interests include image/video generation, editing, representation, and compression.
Guillaume Dauphin received his M.S. degree in electrical engineering from Université Libre de Bruxelles (ULB, French wing of the Brussels University) in 2015. He is now a Ph.D. student at Université Libre de Bruxelles.
Gauthier Lafruit received his master of engineering degree in electronics and Ph.D. degree from the Vrije Universiteit Brussel (Dutch wing of the Brussels University) in 1989 and 1995, respectively. He was a research scientist with the Belgian National Foundation for Scientific Research from 1989 to 1994, being mainly active in the area of wavelet image compression implementations. In 1996, he became the recipient of the Scientific Barco award and joined Imec, Leuven, Belgium, where he was involved as senior scientist with the design of low-power VLSI for combined JPEG/Wavelet compression engines. From 1999 to 2013, Gauthier Lafruit held the position of principal scientist in Multimedia and Vision Systems, Imec. He was then promoted to Professor Visual Computing at Hasselt University before joining l’Université Libre de Bruxelles (French wing of the Brussels University) in 2014, where he is mainly active in the fields of immersive media, as well as virtual and augmented reality. He is currently co-chair of the FTV (free viewpoint TV) working group in the international MPEG standardization committee, and actively contributes to the MPEG-ARAF and JPEGPLENO standardization activities. Prof. Lafruit has published around seventy MPEG/JPEG standardization working documents over the past 15 years, as well as a couple of hundreds of papers in the domain of video and 3D graphics processing.
Adrian Munteanu received his M.S. degree in electronics from Politehnica University of Bucharest, Romania, in 1994, M.S. degree in biomedical engineering from University of Patras, Greece, in 1996, and the doctorate degree in applied sciences (awarded with highest honours) from Vrije Universiteit Brussel (VUB), Belgium, in 2003. In the period 2004–2010 he was a post-doctoral fellow with the Fund for Scientific Research - Flanders (FWO), Belgium, and since 2007, he is a professor at VUB. His research interests include image, video and 3D graphics coding, distributed source coding, error-resilient coding, multimedia transmission over networks, and statistical modeling. A. Munteanu is the author of more than 200 journal and conference publications, book chapters, patent applications, and contributions to standards. He is the recipient of the 2004 BARCO-FWO prize for his Ph.D. work. A. Munteanu currently serves as Associate Editor for IEEE Transactions on Multimedia.
Electronic supplementary material
Rights and permissions
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (https://creativecommons.org/licenses/by/4.0), which permits use, duplication, adaptation, distribution, and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
About this article
Cite this article
Lu, SP., Dauphin, G., Lafruit, G. et al. Color retargeting: Interactive time-varying color image composition from time-lapse sequences. Comp. Visual Media 1, 321–330 (2015). https://doi.org/10.1007/s41095-015-0031-3
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s41095-015-0031-3