Abstract
We present a neural network model approach for multi-frame blind deconvolution. The discriminative approach adopts and combines two recent techniques for image deblurring into a single neural network architecture. Our proposed hybrid-architecture combines the explicit prediction of a deconvolution filter and non-trivial averaging of Fourier coefficients in the frequency domain. In order to make full use of the information contained in all images in one burst, the proposed network embeds smaller networks, which explicitly allow the model to transfer information between images in early layers. Our system is trained end-to-end using standard backpropagation on a set of artificially generated training examples, enabling competitive performance in multi-frame blind deconvolution, both with respect to quality and runtime.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
Provided by [29].
- 2.
References
Burger, H.C., Schuler, C.J., Harmeling, S.: Image denoising: can plain neural networks compete with BM3D? In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE Computer Society (2012)
Schuler, C., Burger, H., Harmeling, S., Scholköpf, B.: A machine learning approach for non-blind image deconvolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE Computer Society (2013)
Delbracio, M., Sapiro, G.: Burst deblurring: removing camera shake through fourier burst accumulation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE Computer Society (2015)
Wang, R., Tao, D.: Recent progress in image deblurring. arXiv preprint (2014). arXiv:1409.6838
Sun, L., Cho, S., Wang, J., Hays, J.: Edge-based blur kernel estimation using patch priors. In: IEEE International Conference in Computational Photography (ICCP), pp. 1–8. IEEE (2013)
Michaeli, T., Irani, M.: Blind deblurring using internal patch recurrence. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8691, pp. 783–798. Springer, Heidelberg (2014). doi:10.1007/978-3-319-10578-9_51
Xu, L., Ren, J.S., Liu, C., Jia, J.: Deep convolutional neural network for image deconvolution. In: Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N.D., Weinberger, K.Q., (eds.): Advances in Neural Information Processing Systems (NIPS), pp. 1790–1798. Curran Associates, Inc. (2014)
Rosenbaum, D., Weiss, Y.: The return of the gating network: combining generative models and discriminative training in natural image priors. In: Advances in Neural Information Processing Systems (NIPS), pp. 2665–2673 (2015)
Schuler, C.J., Hirsch, M., Harmeling, S., Schölkopf, B.: Learning to deblur. IEEE Transactions on Pattern Analysis and Machine Intelligence (2015)
Sun, J., Cao, W., Xu, Z., Ponce, J.: Learning a convolutional neural network for non-uniform motion blur removal. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE Computer Society (2015)
Chakrabarti, A.: A neural approach to blind motion deblurring. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9907, pp. 221–235. Springer, Heidelberg (2016). doi:10.1007/978-3-319-46487-9_14
Hradiš, M., Kotera, J., Zemcík, P., Šroubek, F.: Convolutional neural networks for direct text deblurring. In: Proceedings of BMVC, vol. 10 (2015)
Svoboda, P., Hradi, M., Mark, L., Zemck, P.: CNN for license plate motion deblurring. In: IEEE International Conference on Image Processing (ICIP), pp. 3832–3836 (2016)
Loktyushin, A., Schuler, C., Scheffler, K., Schölkopf, B.: Retrospective motion correction of magnitude-input MR images. In: Bhatia, K.K., Lombaert, H. (eds.) MLMMI 2015. LNCS, vol. 9487, pp. 3–12. Springer, Cham (2015). doi:10.1007/978-3-319-27929-9_1
Zoran, D., Weiss, Y.: From learning models of natural image patches to whole image restoration. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 479–486. IEEE Computer Society (2011)
Hasinoff, S.W., Kutulakos, K.N., Durand, F., Freeman, W.T.: Time-constrained photography. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 333–340. IEEE (2009)
Rav-Acha, A., Peleg, S.: Two motion-blurred images are better than one. Pattern Recogn. Lett. 26, 311–317 (2005)
Zhang, H., Wipf, D.P., Zhang, Y.: Multi-observation blind deconvolution with an adaptive sparse prior. IEEE Trans. Pattern Anal. Mach. Intell. 36, 1628–1643 (2014)
Chen, J., Yuan, L., Tang, C.K., Quan, L.: Robust dual motion deblurring. In: IEEE Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1–8. IEEE Computer Society (2008)
Cai, J.F., Ji, H., Liu, C., Shen, Z.: Blind motion deblurring using multiple images. J. Comput. Phys. 228, 5057–5071 (2009)
Šroubek, F., Milanfar, P.: Robust multichannel blind deconvolution via fast alternating minimization. IEEE Trans. Image Process. 21, 1687–1700 (2012)
Zhu, X., Šroubek, F., Milanfar, P.: Deconvolving PSFs for a better motion deblurring using multiple images. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7576, pp. 636–647. Springer, Heidelberg (2012). doi:10.1007/978-3-642-33715-4_46
Zhang, H., Carin, L.: Multi-shot imaging: joint alignment, deblurring and resolution-enhancement. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE Computer Society (2014)
Zhang, H., Yang, J.: Intra-frame deblurring by leveraging inter-frame camera motion. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE Computer Society (2015)
Kim, T.H., Nah, S., Lee, K.M.: Dynamic scene deblurring using a locally adaptive linear blur model. arXiv preprint (2016). arXiv:1603.04265
Ito, A., Sankaranarayanan, A.C., Veeraraghavan, A., Baraniuk, R.G.: BlurBurst: removing blur due to camera shake using multiple images. ACM Trans. Graph. 3 (2014). Submitted
Garrel, V., Guyon, O., Baudoz, P.: A highly efficient lucky imaging algorithm: image synthesis based on fourier amplitude selection. Publ. Astron. Soc. Pac. 124, 861–867 (2012)
Delbracio, M., Sapiro, G.: Hand-held video deblurring via efficient fourier aggregation. IEEE Trans. Comput. Imaging 1, 270–283 (2015)
Lin, T., Maire, M., Belongie, S.J., Bourdev, L.D., Girshick, R.B., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft COCO: common objects in context. arXiv preprint (2014). arXiv:1405.0312
Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint (2014). arXiv:1412.6980
Duchi, J., Hazan, E., Singer, Y.: Adaptive subgradient methods for online learning and stochastic optimization. J. Mach. Learn. Res. 12, 2121–2159 (2011)
Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G.S., Davis, A., Dean, J., Devin, M., Ghemawat, S., Goodfellow, I., Harp, A., Irving, G., Isard, M., Jia, Y., Jozefowicz, R., Kaiser, L., Kudlur, M., Levenberg, J., Mané, D., Monga, R., Moore, S., Murray, D., Olah, C., Schuster, M., Shlens, J., Steiner, B., Sutskever, I., Talwar, K., Tucker, P., Vanhoucke, V., Vasudevan, V., Viégas, F., Vinyals, O., Warden, P., Wattenberg, M., Wicke, M., Yu, Y., Zheng, X.: TensorFlow: Large-scale machine learning on heterogeneous systems (2015). Software. tensorflow.org
Zhang, H., Wipf, D., Zhang, Y.: Multi-image blind deblurring using a coupled adaptive sparse prior. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1051–1058. IEEE Computer Society (2013)
Köhler, R., Hirsch, M., Mohler, B., Schölkopf, B., Harmeling, S.: Recording and playback of camera shake: benchmarking blind deconvolution with a real-world database. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7578, pp. 27–40. Springer, Heidelberg (2012). doi:10.1007/978-3-642-33786-4_3
Hirsch, M., Sra, S., Schölkopf, B., Harmeling, S.: Efficient filter flow for space-variant multiframe blind deconvolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE (2010)
Pătrăucean, V., Handa, A., Cipolla, R.: Spatio-temporal video autoencoder with differentiable memory. arXiv preprint (2015). arXiv:1511.06309
Jaderberg, M., Simonyan, K., Zisserman, A., et al.: Spatial transformer networks. In: Advances in Neural Information Processing Systems (NIPS), pp. 2008–2016 (2015)
Acknowledgement
This work has been partially supported by the DFG Emmy Noether fellowship Le 1341/1-1 and an NVIDIA hardware grant.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
Wieschollek, P., Schölkopf, B., Lensch, H.P.A., Hirsch, M. (2017). End-to-End Learning for Image Burst Deblurring. In: Lai, SH., Lepetit, V., Nishino, K., Sato, Y. (eds) Computer Vision – ACCV 2016. ACCV 2016. Lecture Notes in Computer Science(), vol 10114. Springer, Cham. https://doi.org/10.1007/978-3-319-54190-7_3
Download citation
DOI: https://doi.org/10.1007/978-3-319-54190-7_3
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-54189-1
Online ISBN: 978-3-319-54190-7
eBook Packages: Computer ScienceComputer Science (R0)