Deep Richardson–Lucy Deconvolution for Low-Light Image Deblurring

Abstract

Images taken under the low-light condition often contain blur and saturated pixels at the same time. Deblurring images with saturated pixels is quite challenging. Because of the limited dynamic range, the saturated pixels are usually clipped in the imaging process and thus cannot be modeled by the linear blur model. Previous methods use manually designed smooth functions to approximate the clipping procedure. Their deblurring processes often require empirically defined parameters, which may not be the optimal choices for different images. In this paper, we develop a data-driven approach to model the saturated pixels by a learned latent map. Based on the new model, the non-blind deblurring task can be formulated into a maximum a posterior problem, which can be effectively solved by iteratively computing the latent map and the latent image. Specifically, the latent map is computed by learning from a map estimation network, and the latent image estimation process is implemented by a Richardson–Lucy (RL)-based updating scheme. To estimate high-quality deblurred images without amplified artifacts, we develop a prior estimation network to obtain prior information, which is further integrated into the RL scheme. Experimental results demonstrate that the proposed method performs favorably against state-of-the-art algorithms both quantitatively and qualitatively on synthetic and real-world images.

Appendix

We present the detailed deviations for solving Eq. (13) in this “Appendix”. By reformulating Eq. (13) into a vectorized form, we can obtain:

$$\begin{aligned} \min _{\textbf {I}} {\textbf {M}}^\text {T}{} {\textbf {KI}} - {\textbf {B}}^\text {T}\log (diag({\textbf {M}}){\textbf {KI}}) + \lambda \overline{{\textbf {1}}}^\text {T} P({\textbf {I}})\overline{{\textbf {1}}}, \end{aligned}$$

(17)

where ${\textbf {M}}$, ${\textbf {B}}$, and ${\textbf {I}}$ denote the vectorized forms of M, B and I; ${\textbf {K}}$ is the Toeplitz matrix of K w.r.t. I; $\overline{{\textbf {1}}}$ denotes a vector whose elements are all ones. For the second term of Eq. (17), we denote it as $\textbf{A}$ and its derivative w.r.t. ${\textbf {I}}$ is:

$$\begin{aligned} \frac{\partial \textbf{A}}{\partial {\textbf {I}}}= & {} \frac{\partial diag({\textbf {M}}){\textbf {KI}}}{\partial {\textbf {I}}}~\frac{\partial log(diag({\textbf {M}}){\textbf {KI}})}{\partial diag({\textbf {M}}){\textbf {KI}}}~\frac{\partial {\textbf {B}}^\text {T}\log (diag({\textbf {M}}){\textbf {KI}})}{\partial log(diag({\textbf {M}}){\textbf {KI}})},\nonumber \\= & {} ({\textbf {K}}^\text {T}diag({\textbf {M}}))diag(\frac{1}{diag({\textbf {M}}){\textbf {KI}}}){\textbf {B}},\nonumber \\= & {} {\textbf {K}}^\text {T}diag(\frac{1}{diag({\textbf {M}}){\textbf {KI}}})(diag({\textbf {M}}){\textbf {B}}), \end{aligned}$$

(18)

where the divide operation is element-wise. Then we can solve Eq. (17) by setting its derivative to zero as:

$$\begin{aligned} {\textbf {K}}^\text {T}{} {\textbf {M}} - {\textbf {K}}^\text {T}diag\left( \frac{1}{diag({\textbf {M}}){\textbf {KI}}}\right) (diag({\textbf {M}}){\textbf {B}}) + \lambda P'({\textbf {I}}) = 0.\nonumber \\ \end{aligned}$$

(19)

Reformulate the above formation into its matrix form, we have:

$$\begin{aligned} M\otimes \widetilde{K} - \frac{M\circ B}{M\circ (I\otimes K)}\otimes \widetilde{K} + \lambda P'_I(I) = 0. \end{aligned}$$

(20)

where $\widetilde{K}$ is the transpose of K that flips the shape of K upside down and left-to-right, $P'_I(I)$ is the first order derivative of $P_I(I)$ w.r.t. I. Recall that the sum of the kernel equals to 1, i.e. $\overline{{\textbf {1}}}^\text {T} \widetilde{{\textbf {K}}} = 1$, where $\widetilde{{\textbf {K}}}$ is the vectorized form of $\widetilde{K}$. Thus, we further have,

$$\begin{aligned}{} & {} M\otimes \widetilde{K} - \frac{M\circ B}{M\circ (I\otimes K)}\otimes \widetilde{K} {+} \lambda P'_I(I) {+} {\textbf {1}} {-} {\textbf {1}}\otimes \widetilde{K} {=} 0,\nonumber \\ \end{aligned}$$

(21)

$$\begin{aligned}{} & {} \left( \frac{B}{I\otimes K} - M + {\textbf {1}}\right) \otimes \widetilde{K} = {\textbf {1}}+\lambda P'_I(I), \end{aligned}$$

(22)

$$\begin{aligned}{} & {} I\circ (\frac{B}{I\otimes K} - M + {\textbf {1}}) \otimes \widetilde{K} = I\circ ({\textbf {1}}+\lambda P'_I(I)), \end{aligned}$$

(23)

where ${\textbf {1}}$ is an all-one image.

In order to solve Eq. (23), we use the fixed point iteration scheme and rewrite it as:

$$\begin{aligned} \frac{I^{t+1}}{I^t} ={\textbf {1}}= \frac{\left( \frac{B}{I^t\otimes K} - M + {\textbf {1}}\right) \otimes \widetilde{K}}{{\textbf {1}}+\lambda P'_I(I^t)}. \end{aligned}$$

(24)

Thus, we can finally get Eq. (14) in our manuscript:

$$\begin{aligned} I^{t+1} = \frac{I^t \circ \left( \left( \frac{B}{I^t\otimes K} - M + {\textbf {1}}\right) \otimes \widetilde{K}\right) }{{\textbf {1}}+\lambda P'_I(I^t)}. \end{aligned}$$

(25)