Skip to main content
SpringerLink
Log in

Image Sequence Interpolation Using Optimal Control

  • Published:
Journal of Mathematical Imaging and Vision Aims and scope Submit manuscript

Abstract

The problem of finding an interpolating image between two given images in an image sequence is considered. The problem is formulated as an optimal control problem governed by a transport equation, i.e. we aim at finding a flow field which transports the first image as close as possible to the second image. This approach bears similarities with the Horn and Schunck method for optical flow calculation but in fact the model is quite different. The images are modeled in the space of functions of bounded variation and an analysis of solutions of transport equations in this space is included. Moreover, the existence of optimal controls is proven and necessary conditions are derived. Finally, two algorithms are given and numerical results are compared with existing methods. The new method is competitive with state-of-the-art methods and even outperforms several existing methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Adams, R.A., Fournier, J.J.: Sobolev Spaces. Academic Press, San Diego (2003)

    MATH  Google Scholar 

  2. Ambrosio, L., Crippa, G., Lellis, C.D., Otto, F., Westdickenberg, M.: Transport Equations and Multi-D Hyperbolic Conservation Laws. Springer, Berlin (2008)

    Book  Google Scholar 

  3. Ambrosio, L., Fusco, N., Pallara, D.: Functions of Bounded Variation and Free Discontinuity Problems. Clarendon Press, Oxford (2000)

    MATH  Google Scholar 

  4. Ambrosio, L., Tilli, P., Zambotti, L.: Introduzione alla teoria della misura ed alla probabilitá. Lecture notes of a course given at the Scuola Normale Superiore, unpublished

  5. Attouch, H., Buttazzo, G., Michaille, G.: Variational Analysis in Sobolev and BV Spaces. SIAM, Philadelphia (2006)

    MATH  Google Scholar 

  6. Aubert, G., Kornprobst, P.: A mathematical study of the relaxed optical flow problem in the space BV(Ω)∗. SIAM J. Math. Anal. 30(6), 1282–1308 (1999)

    Article  MathSciNet  MATH  Google Scholar 

  7. Aubert, G., Kornprobst, P.: Mathematical Problems in Image Processing. Springer, Berlin (2002)

    MATH  Google Scholar 

  8. Baker, S., Scharstein, D., Lewis, J.P., Roth, S., Black, M.J., Szeliski, R.: A database and evaluation methodology for optical flow. In: ICCV, pp. 1–8 (2007)

    Google Scholar 

  9. Barron, J., Khurana, M.: Determining optical flow for large motions using parametric models in a hierarchical framework. In: Vision Interface, pp. 47–56 (1994)

    Google Scholar 

  10. Borzí, A., Ito, K., Kunisch, K.: Optimal control formulation for determining optical flow. SIAM J. Sci. Comput. 24, 818–847 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  11. Bredies, K.: Optimal Control of Degenerate Parabolic Equations in Image Processing. Logos Verlag, Berlin (2008)

    Google Scholar 

  12. Bredies, K.: Weak solutions of linear degenerate parabolic equations and an application in image processing. Commun. Pure Appl. Anal. 8(4), 1203–1229 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  13. Brezzi, F., Fortin, M.: Mixed and Hybrid Finite Element Methods. Springer, Berlin (1991)

    Book  MATH  Google Scholar 

  14. Brox, T., Bruhn, A., Papenberg, N., Weickert, J.: High accuracy optical flow estimation based on a theory for warping. In: Computer Vision—ECCV 2004. Lecture Notes in Computer Science, pp. 25–36. Springer, Berlin (2004)

    Chapter  Google Scholar 

  15. Bruhn, A., Weickert, J., Schnörr, C.: Lucas/Kanade meets Horn/Schunck: combining local and global optical flow methods. Int. J. Comput. Vis. 61(3), 211–231 (2005)

    Article  Google Scholar 

  16. Burt, P.J., Adelson, E.H.: The Laplacian pyramid as a compact image code. IEEE Trans. Commun. 31, 532–540 (1983)

    Article  Google Scholar 

  17. Colombini, F., Luo, T., Rauch, J.: Nearly lipschitzean divergence free transport propagates neither continuity nor BV regularity. Commun. Math. Sci. 2(2), 207–212 (2004)

    MathSciNet  MATH  Google Scholar 

  18. Crippa, G.: The flow associated to weakly differentiable vector fields. Ph.D. thesis, Universität Zürich (2007)

  19. Dang, Q.A.: Using boundary-operator method for approximate solution of a boundary value problem (bvp) for triharmonic equation. Vietnam J. Math. 33(1), 9–18 (2005)

    MathSciNet  MATH  Google Scholar 

  20. DiPerna, R., Lions, J.: Ordinary differential equations, transport theory and Sobolev spaces. Invent. Math. 98, 511–547 (1989)

    Article  MathSciNet  MATH  Google Scholar 

  21. Elman, H., Silvester, D., Wathen, A.: Finite Elements and Fast Iterative Solvers. OXFORD (2005)

    MATH  Google Scholar 

  22. Enkelmann, W.: Investigation of multigrid algorithms for the estimation of optical flow fields in image sequences. Comput. Vis. Graph. Image Process. 43, 150–177 (1998)

    Article  Google Scholar 

  23. Evans, L.C., Gariepy, R.F.: Measure Theory and Fine Properties of Functions. CRC Press, Boca Raton (1992)

    MATH  Google Scholar 

  24. Girault, V., Raviart, P.A.: Finite Element Methods for Navier-Stokes Equations. Springer, Berlin (1986)

    Book  MATH  Google Scholar 

  25. Hartman, P.: Ordinary Differential Equations, 2nd edn. SIAM, Philadelphia (2002)

    Book  MATH  Google Scholar 

  26. Hinterberger, W., Scherzer, O.: Models for image interpolation based on the optical flow. Computing 66, 231–247 (2001)

    Article  MathSciNet  MATH  Google Scholar 

  27. Hirsch, C.: Numerical Computation of Internal & External Flows. Elsevier, Amsterdam (2007)

    Google Scholar 

  28. Horn, B.K., Schunck, B.G.: Determining optical flow. Artif. Intell. 17, 185–203 (1981)

    Article  Google Scholar 

  29. Kameda, Y., Imiya, A.: The William Harvey code: mathematical analysis of optical flow computation for cardiac motion. Comput. Imaging Vis. 36, 81–104 (2007)

    Article  Google Scholar 

  30. Kuzmin, D., Turek, S.: High-resolution FEM-TVD schemes based on a fully multidimensional flux limiter. J. Comput. Phys. 198, 131–158 (2004)

    Article  MathSciNet  MATH  Google Scholar 

  31. Lee, E., Gunzburger, M.: An optimal control formulation of an image registration problem. J. Math. Imaging Vis. 36(1), 69–80 (2010)

    Article  MathSciNet  Google Scholar 

  32. Lions, J.L.: Optimal Control of Systems Governed by Partial Differential Equations. Springer, Berlin (1971)

    MATH  Google Scholar 

  33. Liu, R., Lin, Z., Su, Z.: Learning PDEs for image restoration via optimal control. In: ECCV 2010, vol. 6311, pp. 115–128 (2010)

    Chapter  Google Scholar 

  34. Nagel, H.: Constraints for the estimation of displacement vector fields from image sequences. In: International Joint Conference on Artificial Intelligence, pp. 156–160 (1983)

    Google Scholar 

  35. Press, W.H., Teukolsky, S.A., Vetterling, W.T., Flannery, B.P.: Numerical Recipes, 3rd edn. The Art of Scientific Computing. Cambridge University Press, Cambridge (2007)

    MATH  Google Scholar 

  36. Riley, K.F., Hobson, M.P., Bence, S.J., Bence, S.: Mathematical Methods for Physics and Engineering. Cambridge University Press, Cambridge (2006)

    Google Scholar 

  37. Rudin, L.I., Osher, S.J., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Physica D 60, 259–268 (1992)

    MATH  Google Scholar 

  38. Ruhnau, P., Schnörr, C.: Optical stokes flow estimation: an imaging-based control approach. Exp. Fluids 42(1), 61–78 (2006)

    Article  Google Scholar 

  39. Stich, T., Linz, C., Albuquerque, G., Magnor, M.: View and time interpolation in image space. Pac. Graph. 27(7), 1781–1787 (2008)

    Google Scholar 

  40. Suter, D.: Mixed-finite element based motion estimation. Innov. Technol. Biol. Med. 15(3), 292–307 (1994)

    Google Scholar 

  41. Tröltzsch, F.: Optimale Steuerung partieller Differentialgleichungen. Vieweg, Wiesbaden (2005)

    MATH  Google Scholar 

  42. Watkinson, J.: The MPEG Handbook, 2nd edn. Focal Press, Burlington (2004)

    Google Scholar 

  43. Wedel, A., Pock, T., Zach, C., Bischof, H., Cremers, D.: An improved algorithm for TV-L1 optical flow. In: Statistical and Geometrical Approaches to Visual Motion Analysis: International Dagstuhl Seminar. Revised Papers, Dagstuhl Castle, Germany, July 13–18, 2008, pp. 23–45. Springer, Berlin (2009). doi:10.1007/978-3-642-03061-1_2

    Chapter  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. SCiE, ZeTeM, University of Bremen, Bibliothekstraße 1, 28359, Bremen, Germany

    Kanglin Chen

  2. Institute for Analysis and Algebra, TU Braunschweig, Pockelsstraße 14 - Forum, 38092, Braunschweig, Germany

    Dirk A. Lorenz

Authors
  1. Kanglin Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dirk A. Lorenz
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Dirk A. Lorenz.

Additional information

This work is supported by the Zentrale Forschungsförderung, Universität Bremen within the PhD group “Scientific Computing in Engineering” (SCiE).

Electronic Supplementary Material

Rights and permissions

Reprints and permissions

About this article

Cite this article

Chen, K., Lorenz, D.A. Image Sequence Interpolation Using Optimal Control. J Math Imaging Vis 41, 222–238 (2011). https://doi.org/10.1007/s10851-011-0274-2

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10851-011-0274-2

Keywords

Search

Navigation