Skip to main content
SpringerLink
Log in

A Unified Optimization Perspective to Single/Multi-observation Blur-Kernel Estimation with Applications to Camera-Shake Deblurring and Nonparametric Blind Super-Resolution

  • Published:
Journal of Mathematical Imaging and Vision Aims and scope Submit manuscript

Abstract

The nonparametric blur-kernel estimation, using either single image or multi-observation, has been intensively studied since Fergus et al.’s influential work (ACM Trans Graph 25:787–794, 2006). However, in the current literature there is always a gap between the two highly relevant problems; that is, single- and multi-shot blind deconvolutions are modeled and solved independently, lacking a unified optimization perspective. In this paper, we attempt to bridge the gap between the two problems and propose a rigorous and unified minimization function for single/multi-shot blur-kernel estimation by coupling the maximum-a-posteriori (MAP) and variational Bayesian (VB) principles. The new function is depicted using a directed graphical model, where the sharp image and the inverse noise variance associated with each shot are treated as random variables, while each blur-kernel, in difference from existing VB methods, is just modeled as a deterministic parameter. Utilizing a universal, three-level hierarchical prior on the latent sharp image and a Gamma hyper-prior on each inverse noise variance, single/multi-shot blur-kernel estimation is uniformly formulated as an \({\varvec{\fancyscript{l}}}_{{0.5}}\)-norm-regularized negative log-marginal-likelihood minimization problem. By borrowing ideas of expectation-maximization, majorization-minimization, and mean field approximation, as well as iteratively reweighted least squares, all the unknowns of interest, including the sharp image, the blur-kernels, the inverse noise variances, as well as other relevant parameters are estimated automatically. Compared with most single/multi-shot blur-kernel estimation methods, the proposed approach is not only more flexible in processing multiple observations under distinct imaging scenarios due to its independence of the commutative property of convolution but also more adaptive in sparse image modeling while in the meanwhile with much less implementational heuristics. Finally, the proposed blur-kernel estimation method is naturally applied to two low-level vision problems, i.e., camera-shake deblurring and nonparametric blind super-resolution. Experiments on benchmark real-world motion blurred images, simulated multiple-blurred images, as well as both synthetic and realistic low-resolution blurred images are conducted, demonstrating the superiority of the proposed approach to state-of-the-art single/multi-shot camera-shake deblurring and nonparametric blind super-resolution methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21

Similar content being viewed by others

Notes

  1. I.e., knowing the blur-kernel.

  2. Blur-kernels corresponding to [1, 8, 15–17, 21, 67] are estimated by the Matlab codes provided by the authors.

  3. The authors of [29] integrate both the single- and multi-shot blur-kernel estimations in their Matlab p-codes.

  4. The gradient magnitude image refers to the magnitude image calculated based on the intermediate estimates of horizontal and vertical gradients.

  5. The estimated blur-kernels corresponding to [56] were kindly provided to us by Tomer Michaeli who is the first author of [56].

References

  1. Fergus, R., Singh, B., Hertzmann, A., Roweis, S.T., Freeman, W.T.: Removing camera shake from a single photograph. ACM Trans. Graph. 25(3), 787–794 (2006)

    Article  Google Scholar 

  2. Sroubek, F., Milanfar, P.: Robust multichannel blind deconvolution via fast alternating minimization. IEEE Trans. Image Process. 21(4), 1687–1700 (2012)

    Article  MathSciNet  Google Scholar 

  3. Zhu, X., Milanfar, P.: Restoration for weakly blurred and strongly noisy images. In: IEEE Workshop on Applications of Computer Vision, pp. 103–109 (2011)

  4. Takeda, H., Milanfar, P.: Removing motion blur with space-time processing. IEEE Trans. Image Process. 20(10), 2990–3000 (2011)

    Article  MathSciNet  Google Scholar 

  5. Zhu, X., Cohen, S., Schiller, S., Milanfar, P.: Estimating spatially varying defocus blur from a single image. IEEE Trans. Image Process. 22(12), 4879–4891 (2013)

    Article  MathSciNet  Google Scholar 

  6. Tai, Y.-W., Tan, P., Brown, M.S.: Richardson-Lucy deblurring for scenes under a projective motion path. IEEE Trans. Pattern Anal. Mach. Intell. 33(8), 1603–1618 (2011)

    Article  Google Scholar 

  7. Tai, Y.-W.: Motion-aware noise filtering for deblurring of noisy and blurry images. IEEE CVPR, pp. 17–24 (2012)

  8. Levin, A., Weiss, Y., Durand, F., Freeman, W.T.: Understanding blind deconvolution algorithms. IEEE Trans. Pattern Anal. Mach. Intell. 33(12), 2354–2367 (2011)

    Article  Google Scholar 

  9. Molina, R., Mateos, J., Katsaggelos, A.K.: Blind deconvolution using a variational approach to parameter, image, and blur estimation. IEEE Trans. Image Process. 15(12), 3715–3727 (2006)

    Article  MathSciNet  Google Scholar 

  10. Babacan, S.D., Molina, R., Katsaggelos, A.K.: Variational Bayesian blind deconvolution using a total variation prior. IEEE Trans. Image Process. 18(1), 12–26 (2009)

    Article  MathSciNet  Google Scholar 

  11. Amizic, B., Molina, R., Katsaggelos, A.K.: Sparse Bayesian blind image deconvolution with parameter estimation. EURASIP J. Image Video Process. 2012–20, 1–15 (2012)

    Google Scholar 

  12. Tzikas, D.G., Likas, A.C., Galatsanos, N.P.: The variational approximation for Bayesian inference: life after the EM algorithm. IEEE Signal Process. Mag. 131, 131–146 (2008)

    Article  Google Scholar 

  13. Mohammad-Djafari, A.: Bayesian approach with prior models which enforce sparsity in signal and image processing. EURASIP J. Adv. Signal Process. (Special issue on Sparse Signal Processing) 2012–52, 1–19 (2012)

    Google Scholar 

  14. Mohammad-Djafari, A., Knuth, K.H.: Bayesian approaches. In: Comon, P., Jutten, C., et al. (eds.) Handbook of Blind Source Separation: ICA and Applications, pp. 467–513. Elsevier, New York (2010)

    Chapter  Google Scholar 

  15. Wipf, D.P., Zhang, H.: Analysis of Bayesian blind deconvolution. EMMCVPR. LNCS, vol. 8081, pp. 40–53. Springer, Heidelberg (2013)

    Google Scholar 

  16. Babacan, S.D., Molina, R., Do, M.N., Katsaggelos, A.K.: Bayesian blind deconvolution with general sparse image priors. ECCV. LNCS, vol. 7577, pp. 341–355. Springer, Heidelberg (2012)

    Google Scholar 

  17. Levin, A., Weiss, Y., Durand, F., Freeman, W.T.: Efficient marginal likelihood optimization in blind deconvolution. In: CVPR, pp. 2657–2664 (2011)

  18. Krishnan, D., Tay, T., Fergus, R.: Blind deconvolution using a normalized sparsity measure. In: CVPR pp. 233–240 (2011)

  19. Almeida, M., Almeida, L.: Blind and semi-blind deblurring of natural images. IEEE Trans. Image Process. 19(1), 36–52 (2010)

    Article  MathSciNet  Google Scholar 

  20. Xu, L., Zheng, S., Jia, J.: Unnatural \(L_{0}\) sparse representation for natural image deblurring. In: CVPR, pp. 1107–1114 (2013)

  21. Cho, S., Lee, S.: Fast motion deblurring. ACM Trans. Graph. 28(5): article no. 145 (2009)

  22. Shan, Q., Jia, J., Agarwala, A.: High-quality motion deblurring from a single image. In: SIGGRAPH, ACM, New York, NY, USA, pp. 1–10 (2008)

  23. Xu, L., Jia, J.: Two-phase kernel estimation for robust motion deblurring. ECCV. LNCS, vol. 6311, pp. 157–170. Springer, Heidelberg (2010)

    Google Scholar 

  24. Rav-Acha, A., Peleg, S.: Two motion blurred images are better than one. Pattern Recog. Lett. 26, 311–317 (2005)

    Article  Google Scholar 

  25. Cai, J.-F., Ji, H., Liu, C., Shen, Z.: Blind motion deblurring using multiple images. J. Comput. Phys. 228, 5057–5071 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  26. Zhu, X., Sroubek, F., Milanfar, P.: Deconvolving PSFs for a better motion deblurring using multiple images. ECCV. LNCS, pp. 636–647. Springer, Heidelberg (2012)

    Google Scholar 

  27. Chen, J., Yuan, L., Tang, C.-K., Quan, L.: Robust dual motion deblurring. In: IEEE CVPR, pp. 1–8 (2008)

  28. Harikumar, G., Bresler, Y.: Perfect blind restoration of images blurred by multiple filters: theory and efficient algorithms. IEEE Trans. Image Process. 8(2), 202–219 (1999)

    Article  Google Scholar 

  29. Zhang, H., Wipf, D., Zhang, Y.: Multi-observation blind deconvolution with an adaptive spare prior. IEEE Trans. Pattern Anal. Mach. Intell. 36(8), 1628–1643 (2014)

    Article  Google Scholar 

  30. Yuan, L., Sun, J., Quan, L., Shum, H.-Y.: Image deblurring with blurred/noisy image pairs. ACM Trans. Graph. 26(3), 1–10 (2007)

    Article  MATH  Google Scholar 

  31. Babacan, S.D., Wang, J., Molina, R., Katsaggelos, A.K.: Bayesian blind deconvolution from differently exposed image pairs. IEEE Trans. Image Process. 19(11), 2874–2888 (2009)

    Article  MathSciNet  Google Scholar 

  32. Figueiredo, M., Nowak, R.: A bound optimization approach to wavelet-based image deconvolution. In: ICIP vol. 2, pp. 782–785 (2005)

  33. Freeman, W.T., Pasztor, E.C.: Learning to estimate scenes from images. In: Kearns, M.S., Solla, S.A., Cohn, D.A. (eds.) Adv. Neural Information Processing Systems, Vol. 11, Cambridge University Press, Cambridge. See also http://www.merl.com/reports/TR99-05/ (1999)

  34. Baker, S., Kanade, T.: Hallucinating faces. In: IEEE International Conference on Automatic Face and Gesture Recognition, pp. 83–88 (2000)

  35. Nasrollahi, K., Moeslund, T.B.: Super-resolution: a comprehensive survey. Mach. Vis. Appl. 25(6), 1423–1468 (2014)

    Article  Google Scholar 

  36. Mudenagudi, U., Singla, R., Kalra, P., Banerjee.: Super-resolution using graph-cut. In: ACCV, pp. 385–394 (2006)

  37. Mansouri, M., Mohammad-Djafari, A.: Joint super-resolution and segmentation from a set of low resolution images using a Bayesian approach with a Gauss-Markov-Potts prior. Int. J. Signal and Imaging. Syst. Eng. 3(4), 211–221 (2010)

    Google Scholar 

  38. Farsiu, S., Robinson, D., Elad, M., Milanfar, P.: Advances and challenges in super-resolution. Int. J. Imaging Syst. Technol. 14(2), 47–57 (2004)

    Article  Google Scholar 

  39. Takeda, H., Milanfar, P., Protter, M., Elad, M.: Super-resolution without explicit subpixel motion estimation. IEEE Trans. Image Process. 18(9), 1958–1975 (2009)

    Article  MathSciNet  Google Scholar 

  40. Protter, M., Elad, M., Takeda, H., Milanfar, P.: Generalizing the nonlocal-means to super-resolution reconstruction. IEEE Trans. Image Process. 18(1), 36–51 (2009)

    Article  MathSciNet  Google Scholar 

  41. Shen, H., Zhang, L., Huang, B., Li, P.: A MAP approach for joint motion estimation, segmentation, and super resolution. IEEE Trans. Image Process. 16(2), 479–490 (2007)

    Article  MathSciNet  Google Scholar 

  42. Yang, J., Wright, J., Huang, T., Ma, Y.: Image super-resolution via sparse representation. IEEE TIP 19(11), 2861–2873 (2010)

    MathSciNet  Google Scholar 

  43. Glasner, D., Bagon, S., Irani, M.: Super-resolution from a single image. In: ICCV, pp. 349–356 (2009)

  44. Chang, H., Yeung, D.-Y., Xiong, Y.: Super-resolution through neighbor embedding. In: CVPR, pp. 275–282 (2004)

  45. Zeyde, R., Elad, M., Protter, M.: On single image scale-up using sparse-representations. Curves and Surfaces. LNCS, vol. 6920, pp. 711–730. Springer, Heidelberg (2012)

    Chapter  Google Scholar 

  46. Timofte, R., Smet, V.D., Gool, L.V.: Anchored neighborhood regression for fast exampled-based super-resolution. In: ICCV, pp. 1920–1927 (2013)

  47. Timofte, R., Smet, V.D., Gool, L.V.: A+: Adjusted anchored neighborhood regression for fast super-resolution. In: ECCV, pp. 1–5 (2014)

  48. Dong, C., Loy, C.C., He, K., Tang, X.: Learning a deep convolutional network for image super-resolution. In: ECCV (2014)

  49. Peleg, T., Elad, M.: A statistical prediction model based on sparse representations for single image super-resolution. IEEE Trans. Image Process. 23(6), 2569–2582 (2014)

    Article  MathSciNet  Google Scholar 

  50. Yang, J., Lin, Z., Cohen, S.: Fast image super-resolution based on in-place example regression. In: CVPR ,pp. 1059–1066 (2013)

  51. Fattal, R.: Image upsampling via imposed edge statistics. ACM Trans. Graph. (SIGGRAPH) 26(3), 95–102 (2007)

    Article  Google Scholar 

  52. Efrat, N., Glasner, D., Apartsin, A., Nadler, B., Levin, A.: Accurate blur models vs. image priors in single image super-resolution. In: ICCV (2013)

  53. Begin, I., Ferrie, F.R.: PSF recovery from examples for blind super-resolution. In: ICIP vol. 5, pp. 421–424 (2007)

  54. Wang, Q., Tang, X., Shum, H.: Patch based blind image super resolution. In: ICCV, pp. 709–716 (2005)

  55. Joshi, N., Szeliski, R., Kriegman, D.J.: PSF estimation using sharp edge prediction. In: CVPR, pp. 1–8 (2008)

  56. Michaeli, T., Irani, M.: Nonparametric blind super-resolution. In: ICCV, pp. 945–952 (2013)

  57. Shao, W.-Z., Ge, Q., Deng, H.-S., Wei, Z.-H., Li, H.-B.: Motion deblurring using non-stationary image modeling. J. Math. Imaging Vis. (2014). doi:10.1007/s10851-014-0537-9

  58. Babacan, S.D., Molina, R., Katsaggelos, A.K.: Bayesian compressive sensing using Laplace priors. IEEE Trans. Image Process. 19(1), 53–63 (2010)

    Article  MathSciNet  Google Scholar 

  59. Griffin, J.E., Brown, P.J.: Bayesian adaptive Lassos with non-convex penalization. Aust. NZ. J. Stat. 53, 423–442 (2012)

    Article  MathSciNet  Google Scholar 

  60. Tipping, M.E.: Sparse Bayesian learning and the relevance vector machine. J. Mach. Learn. Res. 1, 211–244 (2001)

    MathSciNet  MATH  Google Scholar 

  61. Krishnan, D., Fergus, R.: Fast image deconvolution using hyper-Laplacian priors. In: NIPS vol. 22, pp. 1033–1041 (2009)

  62. Kheradmand, A., Milanfar, P.: A general framework for regularized, similarity-based image restoration. IEEE Trans. Image Process. 23(12), 5136–5151 (2014)

    Article  MathSciNet  Google Scholar 

  63. Takeda, H., Farsiu, S., Milanfar, P.: Deblurring using regularized locally-adaptive kernel regression. IEEE Trans. Image Process. 17(4), 550–563 (2008)

    Article  MathSciNet  Google Scholar 

  64. Schmidt, U., Rother, C., Nowozin, S., Jancsary, J.: Discriminative non-blind deblurring. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 23–28 (2013)

  65. Schuler, C. J., Burger, H. C., Harmeling, S., Schölkopf, B.: A machine learning approach for non-blind image deconvolution. In: IEEE Conference Computer Vision and Pattern Recognition, pp. 1067–1074 (2013)

  66. Danielyan, A., Katkovnik, V., Egiazarian, K.: BM3D frames and variational image deblurring. IEEE Trans. Image Process. 21(4), 1715–1728 (2012)

    Article  MathSciNet  Google Scholar 

  67. Kotera, J., Sroubek, F., Milanfar, P.: Blind deconvolution using alternating maximum a posteriori estimation with heavy-tailed priors. CAIP, Part II, LNCS, vol. 8048, pp. 59–66. Springer, Heidelberg (2013)

    Google Scholar 

  68. Levin, A., Fergus, R., Durand, F., Freeman, W.T.: Image and depth from a conventional camera with a coded aperture. ACM Trans. Graph. (SIGGRAPH) 26(3), 701–709 (2007)

    Article  Google Scholar 

Download references

Acknowledgments

Many thanks are given to the anonymous reviewers for their insightful comments which help to significantly strengthen the paper. The first author Wen-Ze Shao is very grateful to Prof. Michael Elad for the financial support allowing him to conduct Postdoc research at the Computer Science Department, Technion-Israel Institute of Technology. He also thanks Prof. Michael Zibulevsky from Technion for his help with the numerical implementation of the paper. In addition, many thanks are given to Prof. Yi-Zhong Ma and Dr. Min Wu for their encouraging supports in the past years, as well as Mr. Ya-Tao Zhang and other kind people for helping him through his lost and sad years. This work is supported partially by the National Natural Science Foundation (NSF) of China (Grant No. 61402239), the NSF of Jiangsu Province (Grant No. BK20130868), the NSF for Jiangsu Higher Education Institutions (Grant Nos. 13KJB510022 and 13KJB120005), and the Jiangsu Key Laboratory of Image and Video Understanding for Social Safety (Nanjing University of Science and Technology, Grant No. 30920140122007).

Author information

Authors and Affiliations

  1. College of Telecommunications and Information Engineering, Nanjing University of Posts and Telecommunications, Nanjing, 210003, China

    Wen-Ze Shao, Qi Ge & Hai-Bo Li

  2. Department of Computer Science, Technion-Israel Institute of Technology, 32000, Haifa, Israel

    Wen-Ze Shao

  3. Jiangsu Key Laboratory of Image and Video Understanding for Social Safety, Nanjing University of Science and Technology, Nanjing, 210094, China

    Wen-Ze Shao

  4. School of Science, Nanjing Audit University, Nanjing, 211815, China

    Hai-Song Deng

  5. School of Computer Science and Technology, Nanjing University of Science and Technology, Nanjing, 210094, China

    Zhi-Hui Wei

Authors
  1. Wen-Ze Shao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Qi Ge
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hai-Song Deng
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zhi-Hui Wei
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Hai-Bo Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Wen-Ze Shao.

Appendices

Appendix 1: Derivation of the posterior distributions \(\text {q}(\varsigma _{i} )\) and \(\text {q}(\mathbf {u}_{d} )\)

1.1 Estimating Posterior Distribution \(\hbox {q}(\varsigma _{i})\)

In order to derive the posterior distribution \(\hbox {q}(\varsigma _{i} ),\, -{{\textsf {\textit{F}}}}(\tilde{\text {q}},\{\varvec{\Theta }_{d} \}_{d\in \varvec{\Lambda }},\{\mathbf {k}_{i} \}_{i\in \varvec{\mho }})\) is decomposed with respect to \(\hbox {q}(\varsigma _{i})\) as follow1s:

$$\begin{aligned}&-{{\textsf {\textit{F}}}}(\tilde{\text {q}},\{\varvec{\Theta }_{d} \}_{d\in \varvec{\Lambda }} ,\{\mathbf {k}_{i} \}_{i\in \varvec{\mho }} ) \\&\quad =-\int {\tilde{\text {q}}\log \left( {\frac{\prod \limits _{i\in \varvec{\mho }} {\prod \limits _{d\in \varvec{\Lambda }} {\text {p}(\mathbf {y}_{i\vert d} ,\varvec{\Upsilon }_{d} ,\varsigma _{i} ;\varvec{\Theta }_{d} ,\mathbf {k}_{i} )} } }{\tilde{\text {q}}}} \right) } \\&\qquad \text {d}\{\varvec{\Upsilon }_{d} \}_{d\in \varvec{\Lambda }} \text {d}\{\varsigma _{i} \}_{i\in \varvec{\mho }} \\&\quad =-\int {\tilde{\text {q}}\log \left( {\prod \limits _{i\in \varvec{\mho }} {\prod \limits _{d\in \varvec{\Lambda }} {\text {p}(\mathbf {y}_{i\vert d} ,\varvec{\Upsilon }_{d} ,\varsigma _{i} ;\varvec{\Theta }_{d} ,\mathbf {k}_{i})}}} \right) } \\&\qquad \hbox {d}\{\varvec{\Upsilon }_{d} \}_{d\in \varvec{\Lambda }} \text {d}\{\varsigma _{i} \}_{i\in \varvec{\mho }} +\int {\tilde{\text {q}}\log \tilde{\text {q}}} \text { d}\{\varvec{\Upsilon }_{d} \}_{d\in \varvec{\Lambda }} \text {d}\{\varsigma _{i} \}_{i\in \varvec{\mho }} \\&\quad \triangleq -\int {\prod \limits _{d\in \varvec{\Lambda }} {\text {q}(\varsigma _{i} )} \log \tilde{\hbox {p}}(\{\mathbf {y}_{i\vert d} \}_{d\in \varvec{\Lambda }} ,\varsigma _{i} ;\{\varvec{\Theta }_{d} \}_{d\in \varvec{\Lambda }} ,\mathbf {k}_{i})} \text {d}\varsigma _{i} \\&\qquad +\int {\prod \limits _{i\in \varvec{\mho }} {\prod \limits _{d\in \varvec{\Lambda }} {\text {q}(\varsigma _{i})}} \log \prod \limits _{i\in \varvec{\mho }} {\prod \limits _{d\in \varvec{\Lambda }} {\text {q}(\varsigma _{i} )} } } \hbox {d}\varsigma _{i} \\&\qquad +\int {\prod \limits _{i\in \varvec{\mho }} {\prod \limits _{d\in \varvec{\Lambda }} {\text {q}(\mathbf {u}_{d} )} } \log \prod \limits _{i\in \varvec{\mho }} {\prod \limits _{d\in \varvec{\Lambda }} {\text {q}(\mathbf {u}_{d} )} } } \hbox {d}\{\mathbf {u}_{d} \}_{d\in \varvec{\Lambda }} \\&\qquad +\int {\prod \limits _{i\in \varvec{\mho }} {\prod \limits _{d\in \varvec{\Lambda }} {\text {q}(\varvec{\gamma }_{d})}} \log \prod \limits _{i\in \varvec{\mho }} {\prod \limits _{d\in \varvec{\Lambda }} {\text {q}(\varvec{\gamma }_{d} )} } } \hbox {d}\{\varvec{\gamma }_{d} \}_{d\in \varvec{\Lambda }} \\&\qquad +\int {\prod \limits _{i\in \varvec{\mho }} {\prod \limits _{d\in \varvec{\Lambda }} {\text {q}(\varvec{\beta }_{d} )} } \log \prod \limits _{i\in \varvec{\mho }} {\prod \limits _{d\in \varvec{\Lambda }} {\text {q}(\varvec{\beta }_{d} )} } } \hbox {d}\{\varvec{\beta }_{d} \}_{d\in \varvec{\Lambda }} \\&\quad =-\int {\prod \limits _{d\in \varvec{\Lambda }} {\text {q}(\varsigma _{i} )} \log \tilde{\hbox {p}}(\{\mathbf {y}_{i\vert d} \}_{d\in \varvec{\Lambda }} ,\varsigma _{i} ;\{\varvec{\Theta }_{d} \}_{d\in \varvec{\Lambda }} ,\mathbf {k}_{i})} \hbox {d}\varsigma _{i} \\&\qquad +\int {\prod \limits _{d\in \varvec{\Lambda }} {\text {q}(\varsigma _{i} )} \log \prod \limits _{d\in \varvec{\Lambda }} {\text {q}(\varsigma _{i} )} } \hbox {d}\varsigma _{i} \\&\qquad +\int {\prod \limits _{i'\in \varvec{\mho }\backslash i} {\prod \limits _{d\in \varvec{\Lambda }} {\text {q}(\varsigma _{i'} )} } \log \prod \limits _{i'\in \varvec{\mho }\backslash i} {\prod \limits _{d\in \varvec{\Lambda }} {\text {q}(\varsigma _{i'} )} } } \hbox {d}\varsigma _{i'} \\&\qquad +\int {\prod \limits _{i\in \varvec{\mho }} {\prod \limits _{d\in \varvec{\Lambda }} {\text {q}(\mathbf {u}_{d} )} } \log \prod \limits _{i\in \varvec{\mho }} {\prod \limits _{d\in \varvec{\Lambda }} {\text {q}(\mathbf {u}_{d} )} } } \hbox {d}\{\mathbf {u}_{d} \}_{d\in \varvec{\Lambda }} \\&\qquad +\int {\prod \limits _{i\in \varvec{\mho }} {\prod \limits _{d\in \varvec{\Lambda }} {\text {q}(\varvec{\gamma }_{d})}} \log \prod \limits _{i\in \varvec{\mho }} {\prod \limits _{d\in \varvec{\Lambda }} {\text {q}(\varvec{\gamma }_{d} )} } } \text {d}\{\varvec{\gamma }_{d} \}_{d\in \varvec{\Lambda }} \\&\qquad +\int {\prod \limits _{i\in \varvec{\mho }} {\prod \limits _{d\in \varvec{\Lambda }} {\text {q}(\varvec{\beta }_{d})}} \log \prod \limits _{i\in \varvec{\mho }} {\prod \limits _{d\in \varvec{\Lambda }} {\text {q}(\varvec{\beta }_{d})}}} \text {d}\{\varvec{\beta }_{d} \}_{d\in \varvec{\Lambda }} \end{aligned}$$
$$\begin{aligned}&=\hbox {KL}\left( {\prod \limits _{d\in \varvec{\Lambda }} {\text {q}(\varsigma _{i} )} \vert \vert \tilde{\text {p}}(\{\mathbf {y}_{i\vert d} \}_{d\in \varvec{\Lambda }} ,\varsigma _{i} ;\{\varvec{\Theta }_{d} \}_{d\in \varvec{\Lambda }} ,\mathbf {k}_{i} )} \right) \nonumber \\&\qquad +\int {\prod \limits _{i'\in \varvec{\mho }\backslash i} {\prod \limits _{d\in \varvec{\Lambda }} {\text {q}(\varsigma _{i'} )} } \log \prod \limits _{i'\in \varvec{\mho }\backslash i} {\prod \limits _{d\in \varvec{\Lambda }} {\text {q}(\varsigma _{i'} )} } } \hbox {d}\varsigma _{i'} \nonumber \\&\qquad +\int {\prod \limits _{i\in \varvec{\mho }} {\prod \limits _{d\in \varvec{\Lambda }} {\text {q}(\mathbf {u}_{d} )} } \log \prod \limits _{i\in \varvec{\mho }} {\prod \limits _{d\in \varvec{\Lambda }} {\text {q}(\mathbf {u}_{d} )} } } \hbox {d}\{\mathbf {u}_{d} \}_{d\in \varvec{\Lambda }} \nonumber \\&\qquad +\int {\prod \limits _{i\in \varvec{\mho }} {\prod \limits _{d\in \varvec{\Lambda }} {\text {q}(\varvec{\gamma }_{d} )} } \log \prod \limits _{i\in \varvec{\mho }} {\prod \limits _{d\in \varvec{\Lambda }} {\text {q}(\varvec{\gamma }_{d} )} } } \text {d}\{\varvec{\gamma }_{d} \}_{d\in \varvec{\Lambda }} \nonumber \\&\qquad +\int {\prod \limits _{i\in \varvec{\mho }} {\prod \limits _{d\in \varvec{\Lambda }} {\text {q}(\varvec{\beta }_{d})}} \log \prod \limits _{i\in \varvec{\mho }} {\prod \limits _{d\in \varvec{\Lambda }} {\text {q}(\varvec{\beta }_{d})}}} \text {d}\{\varvec{\beta }_{d} \}_{d\in \varvec{\Lambda }} , \end{aligned}$$
(38)

where

$$\begin{aligned}&\log \tilde{\text {p}}(\{\mathbf {y}_{i\vert d} \}_{d\in \varvec{\Lambda }} ,\varsigma _{i} ;\{\varvec{\Theta }_{d} \}_{d\in \varvec{\Lambda }} ,\mathbf {k}_{i} ) \nonumber \\&\quad = \int \prod \limits _{i'\in \varvec{\mho }\backslash i} {\prod \limits _{d\in \varvec{\Lambda }} {\text {q}(\varsigma _{i'} )\text {q}(\mathbf {u}_{d} )\text {q}(\varvec{\gamma }_{d} )\text {q}(\varvec{\beta }_{d})}} \nonumber \\&\qquad \log \left( {\prod \limits _{i\in \varvec{\mho }} {\prod \limits _{d\in \varvec{\Lambda }} {\text {p}(\mathbf {y}_{i\vert d} ,\varvec{\Upsilon }_{d} ,\varsigma _{i} ;\varvec{\Theta }_{d} ,\mathbf {k}_{i} )} } } \right) \nonumber \\&\qquad \text {d}\{\varvec{\Upsilon }_{d} \}_{d\in \varvec{\Lambda }} \text {d}\{\varsigma _{i'} \}_{i'\in \varvec{\mho }\backslash i} . \end{aligned}$$
(39)

It is seen that, \(-{{\textsf {\textit{F}}}}(\tilde{\text {q}},\{\varvec{\Theta }_{d} \}_{d\in \varvec{\Lambda }} ,\{\mathbf {k}_{i} \}_{i\in \varvec{\mho }} )\) is minimized as the KL divergence \(\hbox {KL}( \prod \nolimits _{d\in \varvec{\Lambda }} {\hbox {q}(\varsigma _{i} )} \vert \vert \tilde{\text {p}}(\{\mathbf {y}_{i\vert d} \}_{d\in \varvec{\Lambda }} ,\varsigma _{i} ;\{\varvec{\Theta }_{d} \}_{d\in \varvec{\Lambda }} ,\mathbf {k}_{i} ) )\) equals zero, i.e., \(\prod \nolimits _{d\in \varvec{\Lambda }} {\text {q}(\varsigma _{i} )} \!=\!\tilde{\text {p}}(\{\mathbf{y}_{i\vert d} \}_{d\in \varvec{\Lambda }} ,\varsigma _{i} ;\{\varvec{\Theta }_{d} \}_{d\in \varvec{\Lambda }} ,\mathbf{k}_{i})\), leading to

$$\begin{aligned} \vert \varvec{\Lambda }\vert \log \text {q}(\varsigma _{i} )= & {} \sum \limits _{d\in \varvec{\Lambda }} {\log \text {q}(\varsigma _{i})} \nonumber \\= & {} \log \tilde{\text {p}}(\{\mathbf {y}_{i\vert d} \}_{d\in \varvec{\Lambda }} ,\varsigma _{i} ;\{\varvec{\Theta }_{d} \}_{d\in \varvec{\Lambda }} ,\mathbf {k}_{i} ) \nonumber \\= & {} \left\langle {\log \prod \limits _{i\in \varvec{\mho }} {\prod \limits _{d\in \varvec{\Lambda }} {\text {p}(\mathbf {y}_{i\vert d} , \varvec{\Upsilon }_{d} ,\varsigma _{i} ;\varvec{\Theta }_{d} ,\mathbf {k}_{i} )} } } \right\rangle _{\{\text {q}(\varsigma _{i'} )\}_{i'\in \varvec{\mho }\backslash i} \{\text {q}(\mathbf {u}_{d} )\}_{d\in \varvec{\Lambda }} \{\text {q}(\varvec{\gamma }_{d} )\}_{d\in \varvec{\Lambda }} \{\text {q}(\varvec{\beta }_{d} )\}_{d\in \varvec{\Lambda }} } \nonumber \\= & {} \left\{ \frac{M}{2}\sum \limits _{d\in \varvec{\Lambda }} {\log \varsigma _{i} } -\frac{\varsigma _{i}}{2}\sum \limits _{d\in \varvec{\Lambda }} {\langle \left\| {\mathbf {y}_{i\vert d} -\mathbf {K}_{i} \mathbf {u}_{d}} \right\| _{2}^{2} \rangle _{\text {q}(\mathbf {u}_{d})}} +\sum \limits _{d\in \varvec{\Lambda }} {\log ((a-1)\log \varsigma _{i} -b\varsigma _{i} )} \right\} +const. \end{aligned}$$
(40)

Therefore, the posterior distribution \(\hbox {q}(\varsigma _{i} )\) is actually a Gamma PDF \({\varvec{\fancyscript{Ga}}}(\varsigma _{i} \vert a_{\varsigma _{i} } ,b_{\varsigma _{i}})\), where the shape and rate parameters are defined as

$$\begin{aligned} a_{\varsigma _{i} }= & {} \frac{M}{2}+a, \end{aligned}$$
(41)
$$\begin{aligned} b_{\varsigma _{i} }= & {} \frac{1}{2\vert \varvec{\Lambda }\vert }\sum \limits _{d\in \varvec{\Lambda }} {\left\langle \left\| {\mathbf {y}_{i\vert d} -\mathbf {K}_{i} \mathbf {u}_{d}} \right\| _{2}^{2} \right\rangle _{\mathrm{q}(\mathbf {u}_{d} )} } +b, \end{aligned}$$
(42)

and its mean is given by \(\langle \varsigma _{i} \rangle _{q(\varsigma _{i} )} =\frac{a_{\varsigma _{i} } }{b_{\varsigma _{i}}}\).

1.2 Estimating Posterior Distribution \(\text {q}(\mathbf {u}_{d})\)

In order to derive \(\text {q}(\mathbf {u}_{d} )\), \(-{{\textsf {\textit{F}}}}(\tilde{\text {q}},\{\varvec{\Theta }_{d} \}_{d\in \varvec{\Lambda }} ,\{\mathbf {k}_{i} \}_{i\in \varvec{\mho }})\) is decomposed with respect to \(\text {q}(\mathbf {u}_{d})\) as follows:

$$\begin{aligned}&-{{\textsf {\textit{F}}}}(\tilde{\text {q}},\{\varvec{\Theta }_{d} \}_{d\in \varvec{\Lambda }} ,\{\mathbf {k}_{i} \}_{i\in \varvec{\mho }} ) \nonumber \\&\quad =-\int {\tilde{\text {q}}\log \left( {\frac{\prod \limits _{i\in \varvec{\mho }} {\prod \limits _{d\in \varvec{\Lambda }} {\text {p}(\mathbf {y}_{i\vert d} ,\varvec{\Upsilon }_{d} ,\varsigma _{i} ;\varvec{\Theta }_{d} ,\mathbf {k}_{i} )} } }{\tilde{\text {q}}}} \right) } \nonumber \\&\qquad \text {d}\{\varvec{\Upsilon }_{d} \}_{d\in \varvec{\Lambda }} \text {d}\{\varsigma _{i} \}_{i\in \varvec{\mho }} \nonumber \\&\quad =-\int {\tilde{\text {q}}\log \left( {\prod \limits _{i\in \varvec{\mho }} {\prod \limits _{d\in \varvec{\Lambda }} {\text {p}(\mathbf {y}_{i\vert d} ,\varvec{\Upsilon }_{d} ,\varsigma _{i} ;\varvec{\Theta }_{d} ,\mathbf {k}_{i} )} } } \right) } \nonumber \\&\qquad \text {d}\{\varvec{\Upsilon }_{d} \}_{d\in \varvec{\Lambda }} \text {d}\{\varsigma _{i} \}_{i\in \varvec{\mho }} +\int {\tilde{\text {q}}\log \tilde{\text {q}}} \text {d}\{\varvec{\Upsilon }_{d} \}_{d\in \varvec{\Lambda }} \text {d}\{\varsigma _{i} \}_{i\in \varvec{\mho }} \nonumber \\&\quad \triangleq -\int {\prod \limits _{i\in \varvec{\mho }} {\text {q}(\mathbf {u}_{d} )} \log \tilde{\text {p}}(\{\mathbf {y}_{i\vert d} \}_{d\in \varvec{\Lambda }} ,\mathbf {u}_{d} ;\{\varvec{\Theta }_{d} \}_{d\in \varvec{\Lambda }} ,\mathbf {k}_{i} )} \text {d}\mathbf {u}_{d}\nonumber \\&\qquad +\int {\prod \limits _{i\in \varvec{\mho }} {\prod \limits _{d\in \varvec{\Lambda }} {\text {q}(\mathbf {u}_{d} )} } \log \prod \limits _{i\in \varvec{\mho }} {\prod \limits _{d\in \varvec{\Lambda }} {\text {q}(\mathbf {u}_{d})}}} \hbox {d}\{\mathbf {u}_{d} \}_{d\in \varvec{\Lambda }} \nonumber \\&\qquad +\int {\prod \limits _{i\in \varvec{\mho }} {\prod \limits _{d\in \varvec{\Lambda }} {\text {q}(\varsigma _{i} )} } \log \prod \limits _{i\in \varvec{\mho }} {\prod \limits _{d\in \varvec{\Lambda }} {\text {q}(\varsigma _{i})}}} \text {d}\varsigma _{i}\nonumber \\&\qquad +\int {\prod \limits _{i\in \varvec{\mho }} {\prod \limits _{d\in \varvec{\Lambda }} {\text {q}(\varvec{\gamma }_{d} )} } \log \prod \limits _{i\in \varvec{\mho }} {\prod \limits _{d\in \varvec{\Lambda }} {\text {q}(\varvec{\gamma }_{d} )} } } \text {d}\{\varvec{\gamma }_{d} \}_{d\in \varvec{\Lambda }} \nonumber \\&\qquad +\int {\prod \limits _{i\in \varvec{\mho }} {\prod \limits _{d\in \varvec{\Lambda }} {\text {q}(\varvec{\beta }_{d} )} } \log \prod \limits _{i\in \varvec{\mho }} {\prod \limits _{d\in \varvec{\Lambda }} {\text {q}(\varvec{\beta }_{d} )} } } \text {d}\{\varvec{\beta }_{d} \}_{d\in \varvec{\Lambda }}\nonumber \\&\quad =-\int {\prod \limits _{i\in \varvec{\mho }} {\text {q}(\mathbf {u}_{d})} \log \tilde{\text {p}}(\{\mathbf {y}_{i\vert d} \}_{d\in \varvec{\Lambda }} ,\mathbf {u}_{d} ;\{\varvec{\Theta }_{d} \}_{d\in \varvec{\Lambda }} ,\mathbf {k}_{i})} \mathrm{d}\mathbf {u}_{d} \nonumber \\&\qquad +\int {\prod \limits _{i\in \varvec{\mho }} {\text {q}(\mathbf {u}_{d} )} \log \prod \limits _{i\in \varvec{\mho }} {\text {q}(\mathbf {u}_{d} )} } \text {d}\mathbf {u}_{d}\nonumber \\&\qquad +\int {\prod \limits _{i\in \varvec{\mho }} {\prod \limits _{d'\in \varvec{\Lambda }\backslash d} {\text {q}(\mathbf {u}_{d'} )} } } \log \prod \limits _{i\in \varvec{\mho }} {\prod \limits _{i\in \varvec{\mho }} {\text {q}(\mathbf {u}_{d'})}} \text {d}\mathbf {u}_{d'} \nonumber \\&\qquad +\int {\prod \limits _{i\in \varvec{\mho }} {\prod \limits _{d\in \varvec{\Lambda }} {\text {q}(\varsigma _{i} )} } \log \prod \limits _{i\in \varvec{\mho }} {\prod \limits _{d\in \varvec{\Lambda }} {\text {q}(\varsigma _{i} )} } } \text {d}\varsigma _{i} \nonumber \\&\qquad +\int {\prod \limits _{i\in \varvec{\mho }} {\prod \limits _{d\in \varvec{\Lambda }} {\text {q}(\varvec{\gamma }_{d} )} } \log \prod \limits _{i\in \varvec{\mho }} {\prod \limits _{d\in \varvec{\Lambda }} {\text {q}(\varvec{\gamma }_{d} )} } } \text {d}\{\varvec{\gamma }_{d} \}_{d\in \varvec{\Lambda }} \nonumber \\&\qquad +\int {\prod \limits _{i\in \varvec{\mho }} {\prod \limits _{d\in \varvec{\Lambda }} {\text {q}(\varvec{\beta }_{d} )} } \log \prod \limits _{i\in \varvec{\mho }} {\prod \limits _{d\in \varvec{\Lambda }} {\text {q}(\varvec{\beta }_{d} )} } } \text {d}\{\varvec{\beta }_{d} \}_{d\in \varvec{\Lambda }} \nonumber \\ \end{aligned}$$
$$\begin{aligned}&\quad =\text {KL}\left( {\prod \limits _{i\in \varvec{\mho }} {\text {q}(\mathbf {u}_{d} )} \vert \vert \tilde{\text {p}}(\{\mathbf {y}_{i\vert d} \}_{d\in \varvec{\Lambda }} ,\mathbf {u}_{d} ;\{\varvec{\Theta }_{d} \}_{d\in \varvec{\Lambda }} ,\mathbf {k}_{i} )} \right) \,\nonumber \\&\qquad +\int {\prod \limits _{i\in \varvec{\mho }} {\prod \limits _{d'\in \varvec{\Lambda }\backslash d} {\text {q}(\mathbf {u}_{d'} )} } } \log \prod \limits _{i\in \varvec{\mho }} {\prod \limits _{i\in \varvec{\mho }} {\text {q}(\mathbf {u}_{d'} )} } \text {d}\mathbf {u}_{d'} \nonumber \\&\qquad +\int {\prod \limits _{i\in \varvec{\mho }} {\prod \limits _{d\in \varvec{\Lambda }} {\text {q}(\varsigma _{i})}} \log \prod \limits _{i\in \varvec{\mho }} {\prod \limits _{d\in \varvec{\Lambda }} {\text {q}(\varsigma _{i} )} } } \text {d}\varsigma _{i}\nonumber \\&\qquad +\int {\prod \limits _{i\in \varvec{\mho }} {\prod \limits _{d\in \varvec{\Lambda }} {\text {q}(\varvec{\gamma }_{d} )} } \log \prod \limits _{i\in \varvec{\mho }} {\prod \limits _{d\in \varvec{\Lambda }} {\text {q}(\varvec{\gamma }_{d} )} } } \text {d}\{\varvec{\gamma }_{d} \}_{d\in \varvec{\Lambda }} \nonumber \\&\qquad +\int {\prod \limits _{i\in \varvec{\mho }} {\prod \limits _{d\in \varvec{\Lambda }} {\text {q}(\varvec{\beta }_{d} )} } \log \prod \limits _{i\in \varvec{\mho }} {\prod \limits _{d\in \varvec{\Lambda }} {\text {q}(\varvec{\beta }_{d} )} } } \text {d}\{\varvec{\beta }_{d} \}_{d\in \varvec{\Lambda }} \end{aligned}$$
(43)

where

$$\begin{aligned}&\log \tilde{\mathrm{p}}(\{ {\mathbf {y}}_{{i|d}} \} _{{d \in {\varvec{\Lambda }}}} ,{\mathbf {u}}_{d} ;\{ {\varvec{\Theta }}_{d} \} _{{d \in {\varvec{\Lambda }}}} ,{\mathbf {k}}_{i} ) \nonumber \\&\quad = \int \prod \limits _{{i \in {\varvec{\mho }}}} {\prod \limits _{{d' \in {\varvec{\Lambda }}\backslash d}} {\text {q}({\mathbf {u}}_{{d'}} )\prod \limits _{{i \in {\varvec{\mho }}}} {\prod \limits _{{d \in {\varvec{\Lambda }}}} {\text {q}(\varsigma _{i} )\text {q}(\varvec{\gamma }_{d} )\text {q}(\varvec{\beta }_{d})}}}} \nonumber \\&\qquad \log \left( {\prod \limits _{{i \in {\varvec{\mho }}}} {\prod \limits _{{d \in {\varvec{\Lambda }}}} {\text {p}({\mathbf {y}}_{{i|d}} ,{\varvec{\Upsilon }}_{d} ,\varsigma _{i} ;{\varvec{\Theta }}_{d} ,{\mathbf {k}}_{i})}}} \right) \nonumber \\&\qquad \text {d}\{ {\mathbf {u}}_{{d'}} \} _{{d' \in {\varvec{\Lambda }}\backslash d}} \text {d}\{ \varvec{\gamma }_{d} \}_{{d \in {\varvec{\Lambda }}}} \text {d}\{\varvec{\beta }_{d} \} _{{d \in {\varvec{\Lambda }}}} \text {d}\{\varsigma _{i} \} _{{i \in {\varvec{\mho }}}}. \end{aligned}$$
(44)

It is seen that, \(-{{\textsf {\textit{F}}}}(\tilde{\text {q}},\{\varvec{\Theta }_{d} \}_{d\in \varvec{\Lambda }} ,\{\mathbf {k}_{i} \}_{i\in \varvec{\mho }} )\) is minimized as the KL divergence \(\hbox {KL}\left( \prod \limits _{i\in \varvec{\mho }} {\text {q}(\mathbf {u}_{d} )} \vert \vert \tilde{\text {p}}(\{\mathbf {y}_{i\vert d} \}_{d\in \varvec{\Lambda }} ,\mathbf {u}_{d} ;\{\varvec{\Theta }_{d} \}_{d\in \varvec{\Lambda }} ,\mathbf {k}_{i} ) \right) \) equals zero, i.e., \(\prod \limits _{i\in \varvec{\mho }} {\text {q}(\mathbf {u}_{d} )} =\tilde{\text {p}}(\{\mathbf {y}_{i\vert d} \}_{d\in {\varvec{\Lambda }} } ,\mathbf {u}_{d} ;\{{\varvec{\Theta }}_{d} \}_{d\in {\varvec{\Lambda }}} ,\mathbf {k}_{i})\), leading to

$$\begin{aligned}&\vert \varvec{\mho }\vert \log \text {q}(\mathbf {u}_{d} )=\sum \limits _{i\in \varvec{\mho }} {\log \text {q}(\mathbf {u}_{d} )} \nonumber \\&\quad =\log \tilde{\text {p}}(\{\mathbf {y}_{i\vert d} \}_{d\in \varvec{\Lambda }} ,\mathbf {u}_{d} ;\{\varvec{\Theta }_{d} \}_{d\in \varvec{\Lambda }} ,\mathbf {k}_{i} ) \nonumber \\&\quad =\left\langle {\log \prod \limits _{i\in \varvec{\mho }} {\prod \limits _{d\in \varvec{\Lambda }} {\text {p}(\mathbf {y}_{i\vert d} ,\varvec{\Upsilon }_{d} ,\varsigma _{i} ;\varvec{\Theta }_{d} ,\mathbf {k}_{i} )} } } \right\rangle _{\{\text {q}(\varsigma _{i} )\}_{i\in \varvec{\mho }} \{\text {q}(\mathbf {u}_{d'} )\}_{d'\in \varvec{\Lambda }\backslash d} \{\text {q}(\varvec{\gamma }_{d} )\}_{d\in \varvec{\Lambda }} \{\text {q}(\varvec{\beta }_{d} )\}_{d\in \varvec{\Lambda }} } \nonumber \\&\quad = -\frac{1}{2}\left\{ \sum \limits _{i\in \varvec{\mho }} {\langle \varsigma _{i} \rangle _{\text {q}(\varsigma _{i} )} \left\| {\mathbf {y}_{i\vert d} -\mathbf {K}_{i} \mathbf {u}_{d}} \right\| _{2}^{2} } +\sum \limits _{i\in \varvec{\mho }} {\mathbf {u}_{d}^{\mathrm{T}} \text {diag}\{\langle \varvec{\gamma }_{d} \rangle _{\text {q}(\varvec{\gamma }_{d} )} \}\mathbf {u}_{d}} \right\} +const. \end{aligned}$$
(45)

Therefore, the posterior distribution \(\text {q}(\mathbf {u}_{d})\) is a multivariate Gaussian PDF \({\varvec{\mathcal {N}}}(\mathbf {u}_{d} \vert \varvec{\mu }_{d} ,\, \mathbf {C}_{d})\), and its mean \(\varvec{\mu }_{d} =\langle \mathbf {u}_{d} \rangle _{\text {q}(\mathbf {u}_{d} )} \) and covariance matrix \(\mathbf {C}_{d}\) are, respectively, defined as

$$\begin{aligned} \varvec{\mu }_{d}= & {} \mathbf {C}_{d} \sum \limits _{i\in \varvec{\mho }} {\langle \varsigma _{i} \rangle _{\mathrm{q}(\varsigma _{i} )} \mathbf {K}_{i}^{\mathrm{T}} \mathbf {y}_{i\vert d} ,} \end{aligned}$$
(46)
$$\begin{aligned} \mathbf {C}_{d}= & {} \left[ {\vert \varvec{\mho } \vert \text {diag}\{\langle \varvec{\gamma }_{d} \rangle _{\mathrm{q}(\varvec{\gamma }_{d})} +\sum \limits _{i\in \varvec{\mho }} {\langle \varsigma _{i} \rangle _{\mathrm{q}(\varsigma _{i} )} \mathbf {K}_{i}^{\mathrm{T}} \mathbf {K}_{i}}} \right] ^{-1}. \end{aligned}$$
(47)

Appendix 2: Representative Dictionary-Based Super-Resolution (SR) Methods

Three representative approaches of dictionary-based single frame fast SR are introduced for reference in Sect. 5.2, i.e., neighborhood embedding (NE) of Chang et al. [44], joint sparse coding (JSC) of Zeyde et al. [45], and anchored neighbor regression (ANR) of Timofte et al. [46]. These three nonblind SR methods all assume the simplest bicubic interpolation kernel in the observation model and are harnessed to generate a super-resolved but blurred image, used as the input of the proposed blur-kernel estimation approach for practicing nonparametric blind SR.

1.1 Neighborhood Embedding

The NE algorithm [44] is a manifold learning approach assuming that image patches of a low-res image and its high-res counterpart form manifolds with similar local geometry in two distinct feature spaces. It roughly means that, as long as enough sampled patches are available, patches in the high-res feature space can be re- constructed as a weighted average of local neighbors using the same weights as in the low-res feature space. In practice, weights are computed by solving a constrained least squares problem.

1.2 Joint Sparse Coding

The JSC algorithm originates from Yang et al. [42], which is improved by Zeyde et al. [45] in both SR restoration quality and execution speed. The core idea of JSC is akin to that of NE. What is distinct to NE is that, the low-res and high-res manifolds in the training step are not the sampled patches but the jointly learned compact dictionaries for the sampled low-res and high-res patches, in order to achieve the same sparse coding for low-res patches as their corresponding high-res patches. Here, the entries of sparse coding play a similar role to the weights in NE. The test high-res patch can be reconstructed by exactly the same sparse coding of its counterpart low-res patch and the learnt high-res dictionary.

1.3 Anchored Neighbor Regression

Combining ideas of NE and JSC, the ANR algorithm [46] achieves comparative or higher quality and one or two orders of magnitude efficiency improvements over the state-of-the-art methods. The ANR is essentially an intimate approximation of the NE based on the JSC, lying in that the low-res and high-res manifolds in the training step are the jointly learned compact dictionaries for the sampled low-res and high-res image patches, just the same as the JSC, but the weights for a test low-res patch are no longer the immediate goals, instead a separate projection matrix to be stored is computed for each atom in the low-res dictionary by Ridge Regression and the neighborhood of the atom. The high-res patch can then be reconstructed using its low-res counterpart and the stored projection matrix corresponding to the atom computed as the nearest neighbor to the test low-res patch.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Shao, WZ., Ge, Q., Deng, HS. et al. A Unified Optimization Perspective to Single/Multi-observation Blur-Kernel Estimation with Applications to Camera-Shake Deblurring and Nonparametric Blind Super-Resolution. J Math Imaging Vis 54, 216–239 (2016). https://doi.org/10.1007/s10851-015-0598-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10851-015-0598-4

Keywords

Search

Navigation