Advertisement

Modeling Blurred Video with Layers

  • Jonas Wulff
  • Michael Julian Black
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8694)

Abstract

Videos contain complex spatially-varying motion blur due to the combination of object motion, camera motion, and depth variation with finite shutter speeds. Existing methods to estimate optical flow, deblur the images, and segment the scene fail in such cases. In particular, boundaries between differently moving objects cause problems, because here the blurred images are a combination of the blurred appearances of multiple surfaces. We address this with a novel layered model of scenes in motion. From a motion-blurred video sequence, we jointly estimate the layer segmentation and each layer’s appearance and motion. Since the blur is a function of the layer motion and segmentation, it is completely determined by our generative model. Given a video, we formulate the optimization problem as minimizing the pixel error between the blurred frames and images synthesized from the model, and solve it using gradient descent. We demonstrate our approach on synthetic and real sequences.

Keywords

Optical Flow Layers Object Boundaries Motion Blur 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Supplementary material

978-3-319-10599-4_16_MOESM1_ESM.pdf (5.5 mb)
Electronic Supplementary Material (PDF 5,605 KB)

References

  1. 1.
    Baker, S., Kanade, T.: Super-resolution optical flow. Tech. Rep. CMU-RI-TR-99-36, Carnegie Mellon University, The Robotics Institute (1999)Google Scholar
  2. 2.
    Bar, L., Berkels, B., Rumpf, M., Sapiro, G.: A variational framework for simultaneous motion estimation and restoration of motion-blurred video. In: IEEE Int. Conf. on Computer Vision (ICCV), pp. 1–8 (2007)Google Scholar
  3. 3.
    Bascle, B., Blake, A., Zisserman, A.: Motion deblurring and super-resolution from an image sequence. In: Buxton, B.F., Cipolla, R. (eds.) ECCV 1996. LNCS, vol. 1065, pp. 573–582. Springer, Heidelberg (1996), http://dl.acm.org/citation.cfm?id=645310.649034 Google Scholar
  4. 4.
    Chakrabarti, A., Zickler, T., Freeman, W.: Analyzing spatially-varying blur. In: IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 2512–2519 (June 2010)Google Scholar
  5. 5.
    Charbonnier, P., Blanc-Feraud, L., Aubert, G., Barlaud, M.: Two deterministic half-quadratic regularization algorithms for computed imaging. In: IEEE Int. Conf. Image Proc. (ICIP), vol. 2, pp. 168–172 (1994)Google Scholar
  6. 6.
    Chen, W.G., Nandhakumar, N., Martin, W.: Image motion estimation from motion smear-a new computational model. IEEE Trans. on Pattern Analysis and Machine Intelligence (PAMI) 18(4), 412–425 (1996)CrossRefGoogle Scholar
  7. 7.
    Cho, S., Lee, S.: Fast motion deblurring. In: ACM Trans. Graphics (TOG) – Proc. SIGGRAPH Asia, pp. 145:1–145:8. ACM, New York (2009), http://doi.acm.org/10.1145/1661412.1618491
  8. 8.
    Cho, S., Wang, J., Lee, S.: Video deblurring for hand-held cameras using patch-based synthesis. ACM Trans. Graph. 31(4), 64:1–64:9 (2012), http://doi.acm.org/10.1145/2185520.2185560
  9. 9.
    Chunhe, S., Hai, Z., Wei, J.: Motion deblurring from a single image using multi-layer statistics priors. In: 2011 IEEE International Conference on Consumer Electronics (ICCE), pp. 481–482 (January 2011)Google Scholar
  10. 10.
    Cremers, D., Soatto, S.: Variational space-time motion segmentation. In: Triggs, B., Zisserman, A. (eds.) Int. Conf. on Computer Vision (ICCV), Nice, vol. 2, pp. 886–892 (October 2003)Google Scholar
  11. 11.
    Dai, S., Wu, Y.: Removing partial blur in a single image. In: IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 2544–2551 (2009)Google Scholar
  12. 12.
    Dai, S., Wu, Y.: Motion from blur. In: IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 1–8 (June 2008)Google Scholar
  13. 13.
    Fergus, R., Singh, B., Hertzmann, A., Roweis, S.T., Freeman, W.T.: Removing camera shake from a single photograph. ACM Trans. Graphics (TOG) – Proc. SIGGRAPH 25(3), 787–794 (2006), http://doi.acm.org/10.1145/1141911.1141956 CrossRefGoogle Scholar
  14. 14.
    Fish, D.A., Brinicombe, A.M., Pike, E.R., Walker, J.G.: Blind deconvolution by means of the richardson–lucy algorithm. J. Opt. Soc. Am. A 12(1), 58–65 (1995), http://josaa.osa.org/abstract.cfm?URI=josaa-12-1-58 CrossRefGoogle Scholar
  15. 15.
    Frey, B., Jojic, N., Kannan, A.: Learning appearance and transparency manifolds of occluded objects in layers. In: IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 45–52 (2003)Google Scholar
  16. 16.
    Grundmann, M., Kwatra, V., Castro, D., Essa, I.: Calibration-free rolling shutter removal. In: IEEE International Conference Computational Photography (ICCP), pp. 1–8 (2012)Google Scholar
  17. 17.
    He, X., Luo, T., Yuk, S.C., Chow, K., Wong, K., Chung, R.H.Y.: Motion estimation method for blurred videos and application of deblurring with spatially varying blur kernels. In: Int. Conf. on Computer Sciences and Convergence Information Technology (ICCIT), pp. 355–359 (2010)Google Scholar
  18. 18.
    Jackson, J., Yezzi, A., Soatto, S.: Dynamic shape and apperance modeling via moving and deforming layers. International Journal of Computer Vision (IJCV) 79(1), 71–84 (2008)CrossRefGoogle Scholar
  19. 19.
    Jepson, A.D., Fleet, D.J., Black, M.J.: A layered motion representation with occlusion and compact spatial support. In: Heyden, A., Sparr, G., Nielsen, M., Johansen, P. (eds.) ECCV 2002, Part I. LNCS, vol. 2350, pp. 692–706. Springer, Heidelberg (2002), http://dx.doi.org/10.1007/3-540-47969-4_46 CrossRefGoogle Scholar
  20. 20.
    Jin, H., Favaro, P., Cipolla, R.: Visual tracking in the presence of motion blur. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2005, vol. 2, pp. 18–25 (2005)Google Scholar
  21. 21.
    Jojic, N., Frey, B.: Learning flexible sprites in video layers. In: IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), vol. 1, pp. I-199–I-206 (2001)Google Scholar
  22. 22.
    Kolmogorov, V., Zabin, R.: What energy functions can be minimized via graph cuts? IEEE Trans. Pattern Analysis and Machine Intelligence (PAMI) 26(2), 147–159 (2004)CrossRefGoogle Scholar
  23. 23.
    Lee, S.H., Moon, N.S., Lee, C.W.: Recovery of blurred video signals using iterative image restoration combined with motion estimation. In: Int. Conf. on Image Processing (ICIP), vol. 1, pp. 755–758 (1997)Google Scholar
  24. 24.
    Levin, A.: Blind motion deblurring using image statistics. In: Schölkopf, B., Platt, J., Hoffman, T. (eds.) Advances in Neural Information Processing Systems, vol. 19, pp. 841–848. MIT Press, Cambridge (2007)Google Scholar
  25. 25.
    Li, Y., Kang, S.B., Joshi, N., Seitz, S., Huttenlocher, D.: Generating sharp panoramas from motion-blurred videos. In: IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 2424–2431 (2010)Google Scholar
  26. 26.
    Lin, H.T., Tai, Y.W., Brown, M.: Motion regularization for matting motion blurred objects. IEEE Trans. Pattern Analysis and Machine Intelligence (PAMI) 33(11), 2329–2336 (2011)CrossRefGoogle Scholar
  27. 27.
    Nir, T., Kimel, R., Bruckstein, A.: Variational approach for joint optic-flow computation and video restoration. Tech. Rep. CIS-2005-03, Technion Israel Institute of Technology (2005)Google Scholar
  28. 28.
    Paramanand, C., Rajagopalan, A.: Non-uniform motion deblurring for bilayer scenes. In: IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 1115–1122 (2013)Google Scholar
  29. 29.
    Pawan Kumar, M., Torr, P., Zisserman, A.: Learning layered motion segmentations of video. International Journal of Computer Vision (IJCV) 76(3), 301–319 (2008), http://dx.doi.org/10.1007/s11263-007-0064-x CrossRefGoogle Scholar
  30. 30.
    Portz, T., Zhang, L., Jiang, H.: Optical flow in the presence of spatially-varying motion blur. In: IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 1752–1759 (2012)Google Scholar
  31. 31.
    Rav-Acha, A., Kohli, P., Rother, C., Fitzgibbon, A.: Unwrap mosaics: A new representation for video editing. ACM Transactions on Graphics (TOG) - Proc. SIGGRAPH 27(3), 17:1–17:11 (2008)Google Scholar
  32. 32.
    Schoenemann, T., Cremers, D.: A coding cost framework for super-resolution motion layer decomposition. IEEE Trans. Im. Proc. 23(3), 1097–1110 (2012)CrossRefMathSciNetGoogle Scholar
  33. 33.
    Seitz, S., Baker, S.: Filter flow. In: 2009 IEEE 12th International Conference on Computer Vision, pp. 143–150 (2009)Google Scholar
  34. 34.
    Sellent, A., Eisemann, M., Goldlucke, B., Cremers, D., Magnor, M.: Motion field estimation from alternate exposure images. IEEE Trans. Pattern Analysis and Machine Intelligence (PAMI) 33(8), 1577–1589 (2011)CrossRefGoogle Scholar
  35. 35.
    Shan, Q., Xiong, W., Jia, J.: Rotational motion deblurring of a rigid object from a single image. In: IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 1–8 (2007)Google Scholar
  36. 36.
    Sun, D., Sudderth, E., Black, M.J.: Layered segmentation and optical flow estimation over time. In: IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 1768–1775 (2012)Google Scholar
  37. 37.
    Sun, D., Wulff, J., Sudderth, E., Pfister, H., Black, M.: A fully-connected layered model of foreground and background flow. In: IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), Portland, OR, pp. 2451–2458 (June 2013)Google Scholar
  38. 38.
    Takeda, H., Milanfar, P.: Removing motion blur with space-time processing. IEEE Transactions on Image Processing 20(10), 2990–3000 (2011)CrossRefMathSciNetGoogle Scholar
  39. 39.
    Torr, P.H.S., Szeliski, R., Anandan, P.: An integrated bayesian approach to layer extraction from image sequences. In: IEEE Int. Conf. on Computer Vision (ICCV), vol. 2, pp. 983–990 (1999)Google Scholar
  40. 40.
    Wang, J.Y.A., Adelson, E.H.: System for encoding image data into multiple layers representing regions of coherent motion and associated motion parameters. US Pat. 5557684 (1996)Google Scholar
  41. 41.
    Wang, J., Adelson, E.: Representing moving images with layers. IEEE Trans. Image Processing 3(5), 625–638 (1994)CrossRefGoogle Scholar
  42. 42.
    Weiss, Y.: Smoothness in layers: Motion segmentation using nonparametric mixture estimation. In: IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), Washington, DC, USA, pp. 520–526 (1997), http://dl.acm.org/citation.cfm?id=794189.794437
  43. 43.
    Whyte, O., Sivic, J., Zisserman, A., Ponce, J.: Non-uniform deblurring for shaken images. International Journal of Computer Vision (IJCV) 98(2), 168–186 (2012)CrossRefzbMATHMathSciNetGoogle Scholar
  44. 44.
    Xu, L., Jia, J., Matsushita, Y.: Motion detail preserving optical flow estimation. IEEE Trans. Pattern Analysis and Machine Intelligence (PAMI) 34(9), 1744–1757 (2012)CrossRefGoogle Scholar
  45. 45.
    Yalcin, H., Black, M.J., Fablet, R.: The dense estimation of motion and appearance in layers. In: IEEE Workshop on Image and Video Registration (June 2004)Google Scholar
  46. 46.
    Yamaguchi, T., Fukuda, H., Furukawa, R., Kawasaki, H., Sturm, P.: Video deblurring and super-resolution technique for multiple moving objects. In: Kimmel, R., Klette, R., Sugimoto, A. (eds.) ACCV 2010, Part IV. LNCS, vol. 6495, pp. 127–140. Springer, Heidelberg (2011), http://dl.acm.org/citation.cfm?id=1966111.1966123 CrossRefGoogle Scholar
  47. 47.
    Zhou, Y., Tao, H.: A background layer model for object tracking through occlusion. In: IEEE Int. Conf. on Computer Vision (CVPR), vol. 2, pp. 1079–1085 (2003)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Jonas Wulff
    • 1
  • Michael Julian Black
    • 1
  1. 1.Max Planck Institute for Intelligent SystemsTübingenGermany

Personalised recommendations