OpenDR: An Approximate Differentiable Renderer

  • Matthew M. Loper
  • Michael J. Black
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8695)

Abstract

Inverse graphics attempts to take sensor data and infer 3D geometry, illumination, materials, and motions such that a graphics renderer could realistically reproduce the observed scene. Renderers, however, are designed to solve the forward process of image synthesis. To go in the other direction, we propose an approximate differentiable renderer (DR) that explicitly models the relationship between changes in model parameters and image observations. We describe a publicly available OpenDR framework that makes it easy to express a forward graphics model and then automatically obtain derivatives with respect to the model parameters and to optimize over them. Built on a new auto-differentiation package and OpenGL, OpenDR provides a local optimization method that can be incorporated into probabilistic programming frameworks. We demonstrate the power and simplicity of programming with OpenDR by using it to solve the problem of estimating human body shape from Kinect depth and RGB data.

Keywords

Inverse graphics Rendering Optimization Automatic Differentiation Software Programming 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Supplementary material

978-3-319-10584-0_11_MOESM1_ESM.pdf (194 kb)
Electronic Supplementary Material (PDF 195 KB)
978-3-319-10584-0_11_MOESM2_ESM.pdf (22.8 mb)
Electronic Supplementary Material (PDF 23,318 KB)
978-3-319-10584-0_11_MOESM3_ESM.mov (5.1 mb)
Electronic Supplementary Material (MOV 5,257 KB)
978-3-319-10584-0_11_MOESM4_ESM.zip (0 kb)
Electronic Supplementary Material (ZIP 7 KB)

References

  1. 1.
    Allen, B., Curless, B., Popović, Z.: The space of human body shapes: Reconstruction and parameterization from range scans. ACM Trans. Graph. 22(3), 587–594 (2003)CrossRefGoogle Scholar
  2. 2.
    Bastian, J.W.: Reconstructing 3D geometry from multiple images via inverse rendering. Ph.D. thesis, University of Adelaide (2008)Google Scholar
  3. 3.
    Baumgart, B.G.: Geometric modeling for computer vision. Tech. Rep. AI Lab Memo AIM-249, Stanford University (Oct 1974)Google Scholar
  4. 4.
    Bergstra, J., Breuleux, O., Bastien, F., Lamblin, P., Pascanu, R., Desjardins, G., Turian, J., Warde-Farley, D., Bengio, Y.: Theano: a CPU and GPU math expression compiler. In: Proceedings of the Python for Scientific Computing Conference (SciPy) (June 2010)Google Scholar
  5. 5.
    Bertero, M., Poggio, T., Torre, V.: Ill-posed problems in early vision. Proc. IEEE 76(8), 869–889 (1988)CrossRefGoogle Scholar
  6. 6.
    Blanz, V., Vetter, T.: A morphable model for the synthesis of 3D faces. In: Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 1999, pp. 187–194. ACM Press/Addison-Wesley Publishing Co., New York (1999)CrossRefGoogle Scholar
  7. 7.
    Bradski, G., Kaehler, A.: Learning OpenCV. O’Reilly Media Inc. (2008), http://oreilly.com/catalog/9780596516130
  8. 8.
    Goodman, N., Mansinghka, V., Roy, D., Bonawitz, K., Tenenbaum, J.: Church: A language for generative models. In: McAllester, D.A., Myllymäki, P. (eds.) Proc. Uncertainty in Artificial Intelligence (UAI), pp. 220–229 (July 2008)Google Scholar
  9. 9.
    Grenander, U.: Lectures in Pattern Theory I, II and III: Pattern Analysis, Pattern Synthesis and Regular Structures. Springer, Heidelberg (1976–1981) Google Scholar
  10. 10.
    Griewank, A.: Evaluating Derivatives: Principles and Techniques of Algorithmic Differentiation. Frontiers in Appl. Math., vol. 19. SIAM, Philadelphia (2000)Google Scholar
  11. 11.
    Heikkila, J., Silven, O.: A four-step camera calibration procedure with implicit image correction. In: Proceedings of 1997 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 1106–1112 (June 1997)Google Scholar
  12. 12.
    Hirshberg, D.A., Loper, M., Rachlin, E., Black, M.J.: Coregistration: Simultaneous alignment and modeling of articulated 3D shape. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012, Part VI. LNCS, vol. 7577, pp. 242–255. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  13. 13.
    Horn, B.: Understanding image intensities. Artificial Intelligence 8, 201–231 (1977)CrossRefMATHGoogle Scholar
  14. 14.
    Jalobeanu, A., Kuehnel, F., Stutz, J.: Modeling images of natural 3D surfaces: Overview and potential applications. In: Conference on Computer Vision and Pattern Recognition Workshop, CVPRW 2004, pp. 188–188 (June 2004)Google Scholar
  15. 15.
    Jampani, V., Nowozin, S., Loper, M., Gehler, P.V.: The informed sampler: A discriminative approach to Bayesian inference in generative computer vision models. CoRR abs/1402.0859 (February 2014), http://arxiv.org/abs/1402.0859
  16. 16.
    Jones, E., Oliphant, T., Peterson, P., et al.: SciPy: Open source scientific tools for Python (2001), http://www.scipy.org/
  17. 17.
    Jones, M.J., Poggio, T.: Model-based matching by linear combinations of prototypes. In: A.I. Memo 1583, pp. 1357–1365. MIT (1996)Google Scholar
  18. 18.
    Kersten, D.: Inverse 3-D graphics: A metaphor for visual perception. Behavior Research Methods, Instruments & Computers 29(1), 37–46 (1997)CrossRefMathSciNetGoogle Scholar
  19. 19.
    Knill, D.C., Richards, W.: Perception as Bayesian Inference. The Press Syndicate of the University of Cambridge, Cambridge (1996)CrossRefMATHGoogle Scholar
  20. 20.
    Gorce, M.d.L., Paragios, N., Fleet, D.J.: Model-based hand tracking with texture, shading and self-occlusions. In: 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2008), Anchorage, Alaska, USA, June 24-26. IEEE Computer Society (2008)Google Scholar
  21. 21.
    Lee, A.D.: ad: a python package for first- and second-order automatic differentation (2012), http://pythonhosted.org/ad/
  22. 22.
    Mansinghka, V., Kulkarni, T.D., Perov, Y.N., Tenenbaum, J.: Approximate Bayesian image interpretation using generative probabilistic graphics programs. In: Burges, C., Bottou, L., Welling, M., Ghahramani, Z., Weinberger, K. (eds.) Advances in Neural Information Processing Systems 26, pp. 1520–1528 (2013)Google Scholar
  23. 23.
    Minka, T., Winn, J., Guiver, J., Knowles, D.: Infer.NET 2.4. Microsoft Research Cambridge (2010), http://research.microsoft.com/infernet.
  24. 24.
    Mumford, D.: Neuronal architectures for pattern-theoretic problems. In: Koch, C., Davis, J.L. (eds.) Large-scale Neuronal theories of the Brain, pp. 125–152. Bradford (1994)Google Scholar
  25. 25.
    Oliphant, T.E.: Python for scientific computing. Computing in Science and Engineering 9(3), 10–20 (2007)CrossRefGoogle Scholar
  26. 26.
    Patow, G., Pueyo, X.: A survey of inverse rendering problems. Computer Graphics Forum 22(4), 663–687 (2003)CrossRefMATHGoogle Scholar
  27. 27.
    Richards, W.: Natural Computation. The MIT Press (A Bradford Book), Cambridge (1988)MATHGoogle Scholar
  28. 28.
    Robinette, K., Blackwell, S., Daanen, H., Boehmer, M., Fleming, S., Brill, T., Hoeferlin, D., Burnsides, D.: Civilian American and European Surface Anthropometry Resource (CAESAR) final report. Tech. Rep. AFRL-HE-WP-TR-2002-0169, US Air Force Research Laboratory (2002)Google Scholar
  29. 29.
    Smelyansky, V.N., Morris, R.D., Kuehnel, F.O., Maluf, D.A., Cheeseman, P.: Dramatic improvements to feature based stereo. In: Heyden, A., Sparr, G., Nielsen, M., Johansen, P. (eds.) ECCV 2002, Part II. LNCS, vol. 2351, pp. 247–261. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  30. 30.
    Sminchisescu, C., Telea, A.: A framework for generic state estimation in computer vision applications. In: International Conference on Computer Vision Systems (ICVS), pp. 21–34 (2001)Google Scholar
  31. 31.
    Sminchisescu, C.: Estimation algorithms for ambiguous visual models: Three dimensional human modeling and motion reconstruction in monocular video sequences. Ph.D. thesis, Inst. National Polytechnique de Grenoble (July 2002)Google Scholar
  32. 32.
    Terzopoulos, D.: Regularization of inverse visual problems involving discontinuities. IEEE PAMI 8(4), 413–424 (1986)CrossRefGoogle Scholar
  33. 33.
    Terzopoulos, D.: Physically-based modeling: Past, present, and future. In: ACM SIGGRAPH 89 Panel Proceedings, pp. 191–209 (1989)Google Scholar
  34. 34.
    Viola, F., Fitzgibbon, A., Cipolla, R.: A unifying resolution-independent formulation for early vision. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 494–501 (June 2012)Google Scholar
  35. 35.
    Walter, S.F.: Pyadolc 2.4.1 (2012), https://github.com/b45ch1/pyadolc
  36. 36.
    Weiss, A., Hirshberg, D., Black, M.: Home 3D body scans from noisy image and range data. In: Int. Conf. on Computer Vision (ICCV), pp. 1951–1958. IEEE, Barcelona (2011)Google Scholar
  37. 37.
    Wood, F., van de Meent, J.W., Mansinghka, V.: A new approach to probabilistic programming inference. In: Artificial Intelligence and Statistics (2014)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Matthew M. Loper
    • 1
  • Michael J. Black
    • 1
  1. 1.Max Planck Institute for Intelligent SystemsTübingenGermany

Personalised recommendations