The Visual Computer

, Volume 33, Issue 1, pp 33–46 | Cite as

Artistic relighting of paintings and drawings

Original Article

Abstract

We present a practical solution to the problem of subject relighting in paintings and drawings. Our interactive technique uses 3-D shading proxies and can be applied to objects with arbitrary geometries. Given a user-provided guess for the shading of an object in a painting/drawing and its corresponding target shading, we refine them using shading-color correlation and a multi-scale scheme. These refined shadings are then used to create a multi-channel shading-ratio image to perform relighting, while taking into account the colors used by the artists to convey shading information. We demonstrate the effectiveness of our solution on a variety of artistic styles, including paintings with strong brush strokes and unconventional shading encodings, drawings, and other types of artwork. Our method is the first to perform relighting of paintings and drawings and, in addition to relighting, can transfer smooth normal and depth maps from 3-D proxies to images.

Keywords

Image relighting Painting relighting Drawing relighting Normal and depth map transfer 

References

  1. 1.
    Agarwala, A., Dontcheva, M., Agrawala, M., Drucker, S., Colburn, A., Curless, B., Salesin, D., Cohen, M.: Interactive digital photomontage. ACM TOG 23(3), 294–302 (2004)CrossRefGoogle Scholar
  2. 2.
    Akers, D., Losasso, F., Klingner, J., Agrawala, M., Rick, J., Hanrahan, P.: Conveying shape and features with image-based relighting. In: VIS’03, pp. 46–51. IEEE Computer Society, Washington, DC (2003)Google Scholar
  3. 3.
    Anrys, F., Dutré, P., Willems, Y.D.: Image-based lighting design. In: Proceedings of the 4th IASTED international conference on visualization, imaging, and image processing (2004)Google Scholar
  4. 4.
    Anthropics: PortraitPro v12.4–Trial Version (2015). http://www.portraitprofessional.com/. Accessed June 2015
  5. 5.
    Barron, J.T., Malik, J.: Color constancy, intrinsic images, and shape estimation. In: ECCV’12, pp. 57–70. Springer, Berlin, Heidelberg (2012)Google Scholar
  6. 6.
    Barron, J.T., Malik, J.: Shape, illumination, and reflectance from shading. IEEE Trans Pattern Anal Mach Intell 37(8), 1670–1687 (2015). doi:10.1109/TPAMI.2014.2377712
  7. 7.
    Basso, A., Graf, H., Gibbon, D., Cosatto, E., Liu, S.: Virtual light: digitally-generated lighting for video conferencing applications. In: ICIP’01, vol. 2, pp. 1085–1088 (2001)Google Scholar
  8. 8.
    Blanz, V., Vetter, T.: A morphable model for the synthesis of 3d faces. In: SIGGRAPH ’99, pp. 187–194. ACM Press, Addison-Wesley Publishing Co, New York (1999)Google Scholar
  9. 9.
    Boivin, S., Gagalowicz, A.: Inverse rendering from a single image. In: CGIV’02, pp. 268–277 (2002)Google Scholar
  10. 10.
    Chen, X., Chen, M., Jin, X., Zhao, Q.: Face illumination transfer through edge-preserving filters. In: CVPR ’11, pp. 281–287. IEEE (2011)Google Scholar
  11. 11.
    Chen, T., Zhu, Z., Shamir, A., Hu, S.M., Cohen-Or, D.: 3Sweep: extracting editable objects from a single photo. ACM TOG 32(6), 195:1–195:10 (2013)Google Scholar
  12. 12.
    Chen, X., Jin, X., Zhao, Q., Wu, H.: Artistic illumination transfer for portraits. CGF 31(4), 1425–1434 (2012)Google Scholar
  13. 13.
    Costa, A., Sousa, A., Nunes Ferreira, F.: Lighting design: A goal based approach using optimisation. In: EGWR’99, pp. 317–328. Springer, Vienna (1999)Google Scholar
  14. 14.
    CrazyBump: Demo Version 1.2 (2014). http://www.crazybump.com/. Accessed Jan 2015
  15. 15.
    Debevec, P., Hawkins, T., Tchou, C., Duiker, H.P., Sarokin, W., Sagar, M.: Acquiring the reflectance field of a human face. In: SIGGRAPH ’00, pp. 145–156. ACM Press, Addison-Wesley Publishing Co, New York (2000)Google Scholar
  16. 16.
    Hu, K., Liu, Y., Dong, Q., Liu, H., Xing, G.: Color face image decomposition under complex lighting conditions. Vis. Comput. 30(6–8), 685–695 (2014)CrossRefGoogle Scholar
  17. 17.
    Johnston, S.F.: Lumo: Illumination for cel animation. In: NPAR ’02, pp. 45-ff. ACM, New York (2002)Google Scholar
  18. 18.
    Kholgade, N., Simon, T., Efros, A., Sheikh, Y.: 3d object manipulation in a single photograph using stock 3d models. ACM TOG 33(4), 127:1–127:12 (2014)Google Scholar
  19. 19.
    Li, Q., Yin, W., Deng, Z.: Image-based face illumination transferring using logarithmic total variation models. Vis. Comput. 26(1), 41–49 (2009)CrossRefGoogle Scholar
  20. 20.
    Li, G., Wu, C., Stoll, C., Liu, Y., Varanasi, K., Dai, Q., Theobalt, C.: Capturing relightable human performances under general uncontrolled illumination. CGF 32(2), 275–284 (2013)Google Scholar
  21. 21.
    Lipman, Y., Levin, D., Cohen-Or, D.: Green coordinates. ACM TOG 27(3), 78:1–78:10 (2008)Google Scholar
  22. 22.
    Lopez-Moreno, J., Jimenez, J., Hadap, S., Reinhard, E., Anjyo, K., Gutierrez, D.: Stylized depiction of images based on depth perception. In: NPAR ’10. ACM, New York (2010)Google Scholar
  23. 23.
    Loscos, C., Drettakis, G., Robert, L.: Interactive virtual relighting of real scenes. IEEE TVCG 6(3), 289–305 (2000)Google Scholar
  24. 24.
    Malzbender, T., Gelb, D., Wolters, H.J.: Polynomial texture maps. In: SIGGRAPH’01, pp. 519–528. ACM, New York (2001)Google Scholar
  25. 25.
    Marschner, S.R.: Inverse rendering for computer graphics. Ph.D. thesis, Cornell University (1998)Google Scholar
  26. 26.
    Mortensen, E.N., Barrett, W.A.: Intelligent scissors for image composition. In: SIGGRAPH ’95, pp. 191–198. ACM, New York (1995)Google Scholar
  27. 27.
    Okabe, M., Zeng, G., Matsushita, Y., Igarashi, T., Quan, L., yeung Shum, H.: Single-view relighting with normal map painting. In: Pacific Graphics, pp. 27–34 (2006)Google Scholar
  28. 28.
    Peers, P., Tamura, N., Matusik, W., Debevec, P.: Post-production facial performance relighting using reflectance transfer. ACM TOG 26(3), 52:1–52:10 (2007)Google Scholar
  29. 29.
    Reinhard, E., Adhikhmin, M., Gooch, B., Shirley, P.: Color transfer between images. IEEE Comput. Graph. Appl. 21(5), 34–41 (2001)CrossRefGoogle Scholar
  30. 30.
    Shapira, D., Avidan, S., Hel-Or, Y.: Multiple histogram matching. In: Proc. IEEE ICIP, pp. 2269–2273 (2013)Google Scholar
  31. 31.
    Shashua, A., Riklin-Raviv, T.: The quotient image: Class-based re-rendering and recognition with varying illuminations. IEEE TPAMI 23(2), 129–139 (2001)CrossRefGoogle Scholar
  32. 32.
    Shi, J., Tomasi, C.: Good features to track. In: CVPR’94, pp. 593–600. IEEE (1994)Google Scholar
  33. 33.
    Shih, Y., Paris, S., Barnes, C., Freeman, W.T., Durand, F.: Style transfer for headshot portraits. ACM TOG 33(4), 148:1–148:14 (2014)Google Scholar
  34. 34.
    Suzuki, S., Abe, K.: Topological structural analysis of digitized binary images by border following. CVGIP 30(1), 32–46 (1985)MATHGoogle Scholar
  35. 35.
    Sýkora, D., Sedlacek, D., Jinchao, S., Dingliana, J., Collins, S.: Adding depth to cartoons using sparse depth (in)equalities. CGF 29(2), 615–623 (2010)Google Scholar
  36. 36.
    Sýkora, D., Kavan, L., Čadík, M., Jamriška, O., Jacobson, A., Whited, B., Simmons, M., Sorkine-Hornung, O.: Ink-and-ray: Bas-relief meshes for adding global illumination effects to hand-drawn characters. ACM TOG 33(2), 16 (2014)CrossRefGoogle Scholar
  37. 37.
    Tunwattanapong, B., Debevec, P.: Interactive image-based relighting with spatially-varying lights. In: ACM SIGGRAPH’09 Poster (2009)Google Scholar
  38. 38.
    Tunwattanapong, B., Ghosh, A., Debevec, P.: Practical image-based relighting and editing with spherical-harmonics and local lights. In: CVMP’11, pp. 138–147. IEEE Computer Society, Washington, DC (2011)Google Scholar
  39. 39.
    Wang, Y., Liu, Z., Hua, G., Wen, Z., Zhang, Z., Samaras, D.: Face re-lighting from a single image under harsh lighting conditions. In: CVPR’07, pp. 1–8. IEEE (2007)Google Scholar
  40. 40.
    Wang, O., Davis, J., Chuang, E., Rickard, I., de Mesa, K., Dave, C.: Video relighting using infrared illumination. CGF 27(2), 271–279 (2008)Google Scholar
  41. 41.
    Welsh, T., Ashikhmin, M., Mueller, K.: Transferring color to greyscale images. ACM TOG 21(3), 277–280 (2002)CrossRefGoogle Scholar
  42. 42.
    Wen, Z., Liu, Z., Huang, T.S.: Face relighting with radiance environment maps. In: CVPR’03, pp. 158–165. IEEE (2003)Google Scholar
  43. 43.
    Wu, T.P., Sun, J., Tang, C.K., Shum, H.Y.: Interactive normal reconstruction from a single image. ACM TOG 27(5), 119:1–119:9 (2008)Google Scholar
  44. 44.
    Wu, T.P., Tang, C.K., Brown, M.S., Shum, H.Y.: Shapepalettes: interactive normal transfer via sketching. ACM TOG 26(3), 44:1–44:5 (2007)Google Scholar
  45. 45.
    Wu, C., Varanasi, K., Liu, Y., Seidel, H.P., Theobalt, C.: Shading-based dynamic shape refinement from multi-view video under general illumination. In: IEEE ICCV, pp. 1108–1115. IEEE (2011)Google Scholar
  46. 46.
    Wu, C., Wilburn, B., Matsushita, Y., Theobalt, C.: High-quality shape from multi-view stereo and shading under general illumination. In: IEEE CVPR, pp. 969–976. IEEE Computer Society, Washington, DC (2011)Google Scholar
  47. 47.
    Yu, T., Wang, H., Ahuja, N., Chen, W.C.: Sparse lumigraph relighting by illumination and reflectance estimation from multi-view images. In: EGSR, pp. 41–50. Eurographics Association, Aire-la-Ville (2006)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2015

Authors and Affiliations

  1. 1.Instituto de InformáticaUFRGSPorto AlegreBrazil

Personalised recommendations