Variational Networks: Connecting Variational Methods and Deep Learning

  • Erich KoblerEmail author
  • Teresa Klatzer
  • Kerstin Hammernik
  • Thomas Pock
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10496)


In this paper, we introduce variational networks (VNs) for image reconstruction. VNs are fully learned models based on the framework of incremental proximal gradient methods. They provide a natural transition between classical variational methods and state-of-the-art residual neural networks. Due to their incremental nature, VNs are very efficient, but only approximately minimize the underlying variational model. Surprisingly, in our numerical experiments on image reconstruction problems it turns out that giving up exact minimization leads to a consistent performance increase, in particular in the case of convex models.



We acknowledge grant support from the Austrian Science Fund (FWF) under the START project BIVISION, No. Y729 and the European Research Council under the Horizon 2020 program, ERC starting grant HOMOVIS, No. 640156.

Supplementary material

440987_1_En_23_MOESM1_ESM.pdf (1.5 mb)
Supplementary material 1 (pdf 1517 KB)


  1. 1.
    Alexandre, B.: Sharesnet: reducing residual network parameter number by sharing weights. arXiv e-prints 1702.08782 (2017)
  2. 2.
    Bertalmio, M., Sapiro, G., Randall, G.: Morphing active contours. TPAMI 22(7), 733–737 (2000)CrossRefGoogle Scholar
  3. 3.
    Bertsekas, D.P.: Incremental proximal methods for large scale convex optimization. Math. Program. 129(2), 163 (2011). doi: 10.1007/s10107-011-0472-0 MathSciNetCrossRefzbMATHGoogle Scholar
  4. 4.
    Black, M.J., Anandan, P.: The robust estimation of multiple motions: parametric and piecewise-smooth flow fields. Comput. Vis. Image Underst. 63(1), 75–104 (1996)CrossRefGoogle Scholar
  5. 5.
    Blake, A., Zisserman, A.: Visual Reconstruction. MIT Press, Cambridge (1987)Google Scholar
  6. 6.
    Boyd, S., Vandenberghe, L.: Convex Optimization. Cambridge University Press, Cambridge (2004)CrossRefzbMATHGoogle Scholar
  7. 7.
    Bredies, K., Kunisch, K., Pock, T.: Total generalized variation. SIIMS 3(3), 492–526 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  8. 8.
    Chambolle, A., Lions, P.L.: Image recovery via total variation minimization and related problems. Numer. Math. 76(2), 167–188 (1997)MathSciNetCrossRefzbMATHGoogle Scholar
  9. 9.
    Chan, T.F., Vese, L.A.: Active contours without edges. IEEE Trans. Image Process. 10(2), 266–277 (2001)CrossRefzbMATHGoogle Scholar
  10. 10.
    Chen, Y., Ranftl, R., Pock, T.: Insights into analysis operator learning: from patch-based sparse models to higher order MRFs. IEEE Trans. Image Process. 23(3), 1060–1072 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
  11. 11.
    Chen, Y., Yu, W., Pock, T.: On learning optimized reaction diffusion processes for effective image restoration. In: CVPR (2015)Google Scholar
  12. 12.
    Dabov, K., Foi, A., Katkovnik, V.: Image denoising by sparse 3D transformation-domain collaborative filtering. IEEE Trans. Image Process. 16(8), 1–16 (2007)MathSciNetCrossRefGoogle Scholar
  13. 13.
    Domke, J.: Generic methods for optimization-based modeling. In: AISTATS, pp. 318–326 (2012)Google Scholar
  14. 14.
    Freedman, D., Zhang, T.: Active contours for tracking distributions. IEEE Trans. Image Process. 13(4), 518–526 (2004)CrossRefGoogle Scholar
  15. 15.
    Gregor, K., LeCun, Y.: Learning fast approximations of sparse coding. In: ICML (2010)Google Scholar
  16. 16.
    Hammernik, K., Knoll, F., Sodickson, D., Pock, T.: Learning a variational model for compressed sensing MRI reconstruction. In: ISMRM (2016)Google Scholar
  17. 17.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition (2016)Google Scholar
  18. 18.
    Horn, B., Schunck, B.: Determining optical flow. Artif. Intell. 17, 185–203 (1981)CrossRefGoogle Scholar
  19. 19.
    Klatzer, T., Hammernik, K., Knöbelreiter, P., Pock, T.: Learning joint demosaicing and denoising based on sequential energy minimization. In: ICCP (2016)Google Scholar
  20. 20.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: NIPS (2012)Google Scholar
  21. 21.
    Kunisch, K., Pock, T.: A bilevel optimization approach for parameter learning in variational models. SIIMS 6, 938–983 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
  22. 22.
    LeCun, Y., Boser, B., Denker, J.S., Henderson, D., Howard, R.E., Hubbard, W., Jackel, L.D.: Backpropagation applied to handwritten zip code recognition. Neural Comput. 1(4), 541–551 (1989)CrossRefGoogle Scholar
  23. 23.
    Lee, A.B., Pedersen, K.S., Mumford, D.: The nonlinear statistics of high-contrast patches in natural images. IJCV 54(1–3), 83–103 (2003)CrossRefzbMATHGoogle Scholar
  24. 24.
    Levin, A., Weiss, Y., Durand, F., Freeman, W.T.: Understanding and evaluating blind deconvolution algorithms. In: CVPR (2009)Google Scholar
  25. 25.
    Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). doi: 10.1007/978-3-319-10602-1_48 Google Scholar
  26. 26.
    Mao, X.J., Shen, C., Yang, Y.B.: Image restoration using convolutional auto-encoders with symmetric skip connections. arXiv e-prints 1606.08921 (2016)
  27. 27.
    Martin, D., Fowlkes, C., Tal, D., Malik, J.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: ICCV (2001)Google Scholar
  28. 28.
    Masoud, A., Saeid, N.: Multi-residual networks. arXiv e-prints 1609.05672 (2016)
  29. 29.
    Mumford, D., Shah, J.: Optimal approximations by piecewise smooth functions and associated variational problems. Commun. Pure Appl. Math. 42(5), 577–685 (1989)MathSciNetCrossRefzbMATHGoogle Scholar
  30. 30.
    Nedić, A., Bertsekas, D.: Convergence rate of incremental subgradient algorithms. In: Uryasev, S., Pardalos, P.M. (eds.) Stochastic Optimization: Algorithms and Applications. Applied Optimization, vol. 54, pp. 223–264. Springer, Boston (2001)Google Scholar
  31. 31.
    Roth, S., Black, M.J.: Fields of experts. IJCV 82, 205–229 (2009)CrossRefGoogle Scholar
  32. 32.
    Rudin, L.I., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Phys. D: Nonlinear Phenom. 60(1–4), 259–268 (1992)MathSciNetCrossRefzbMATHGoogle Scholar
  33. 33.
    Rumelhart, D.E., Hinton, G.E., Williams, R.J.: Learning representations by back-propagating errors. Nature 323(6088), 533–536 (1986)CrossRefzbMATHGoogle Scholar
  34. 34.
    Schelten, K., Nowozin, S., Jancsary, J., Rother, C., Roth, S.: Interleaved regression tree field cascades for blind image deconvolution. In: IEEE Winter Conference on Applications of Computer Vision (2015)Google Scholar
  35. 35.
    Schmidt, U., Roth, S.: Shrinkage fields for effective image restoration. In: CVPR (2014)Google Scholar
  36. 36.
    Shan, Q., Jia, J., Agarwala, A.: High-quality motion deblurring from a single image. In: SIGGRAPH (2008)Google Scholar
  37. 37.
    Sra, S.: Scalable nonconvex inexact proximal splitting. In: NIPS (2012)Google Scholar
  38. 38.
    Veit, A., Wilber, M., Belongie, S.: Residual networks are exponential ensembles of relatively shallow networks. arXiv e-prints 1605.06431 (2016)
  39. 39.
    Xu, L., Zheng, S., Jia, J.: Unnatural L0 sparse representation for natural image deblurring. In: CVPR (2013)Google Scholar
  40. 40.
    Yu, W., Heber, S., Pock, T.: Learning reaction-diffusion models for image inpainting. In: Gall, J., Gehler, P., Leibe, B. (eds.) GCPR 2015. LNCS, vol. 9358, pp. 356–367. Springer, Cham (2015). doi: 10.1007/978-3-319-24947-6_29 CrossRefGoogle Scholar
  41. 41.
    Zhang, K., Zuo, W., Chen, Y., Meng, D., Zhang, L.: Beyond a Gaussian denoiser: residual learning of deep CNN for image denoising. arXiv e-prints 1608.03981 (2016)
  42. 42.
    Zhao, H., Gallo, O., Frosio, I., Kautz, J.: Loss functions for neural networks for image processing. arXiv e-prints 1511.08861 (2015)
  43. 43.
    Zhu, S.C., Mumford, D.: Prior learning and gibbs reaction-diffusion. TPAMI 19(11), 1236–1250 (1997)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Erich Kobler
    • 1
    Email author
  • Teresa Klatzer
    • 1
  • Kerstin Hammernik
    • 1
  • Thomas Pock
    • 1
    • 2
  1. 1.Institute of Computer Graphics and VisionGraz University of TechnologyGrazAustria
  2. 2.Center for Vision, Automation and ControlAustrian Institute of TechnologyViennaAustria

Personalised recommendations