Efficient Burst Raw Denoising with Variance Stabilization and Multi-frequency Denoising Network

Abstract

With the growing popularity of smartphones, capturing high-quality images is of vital importance to smartphones. The cameras of smartphones have small apertures and small sensor cells, which lead to the noisy images in low light environment. Denoising based on a burst of multiple frames generally outperforms single frame denoising but with the larger compututional cost. In this paper, we propose an efficient yet effective burst denoising system. We adopt a three-stage design: noise prior integration, multi-frame alignment and multi-frame denoising. First, we integrate noise prior by pre-processing raw signals into a variance-stabilization space, which allows using a small-scale network to achieve competitive performance. Second, we observe that it is essential to adopt an explicit alignment for burst denoising, but it is not necessary to integrate an learning-based method to perform multi-frame alignment. Instead, we resort to a conventional and efficient alignment method and combine it with our multi-frame denoising network. At last, we propose a denoising strategy that processes multiple frames sequentially. Sequential denoising avoids filtering a large number of frames by decomposing multiple frames denoising into several efficient sub-network denoising. As for each sub-network, we propose an efficient multi-frequency denoising network to remove noise of different frequencies. Our three-stage design is efficient and shows strong performance on burst denoising. Experiments on synthetic and real raw datasets demonstrate that our method outperforms state-of-the-art methods, with less computational cost. Furthermore, the low complexity and high-quality performance make deployment on smartphones possible.

Appendices

Appendix A Noise Modeling of CMOS Signals

We provide the detailed noise modeling of CMOS signals to obtain the relation between sensor gain and $\sigma _r, \sigma _s$. We define the observed intensity as x and underlying true intensity as $x^{*}$. Following Wang et al. (2020), the raw signal is modeled as

$$\begin{aligned} x \sim q_e \alpha \mathcal {P}\left( \frac{x^{*}}{q_e \alpha }\right) + \mathcal {N}(0, \alpha ^2 \sigma _0^2 + \sigma _{adc}^2), \end{aligned}$$

(A.1)

where $q_e$ is quantum efficiency factor, $\alpha$ is the sensor gain, $\sigma _0$ is the variance of read noise caused by sensor readout effects and $\sigma _{adc}$ is the variance of amplifier noise. Then we have:

$$\begin{aligned} \begin{aligned} \sigma _s&= q_e a \\ \sigma _r^2&= \alpha ^2\sigma _{0}^2 + \sigma _{adc}^2. \end{aligned} \end{aligned}$$

(A.2)

For one fixed senor, $q_e$, $\sigma _{0}$, $\sigma _{adc}$ is unchanged. Then sensor gain $\alpha$ is the only factor to affect $\sigma _s, \sigma _r$.

Fig. 7

Visualization of different variance stabilization transformations. x is the mean of the signal in Poisson distribution. Var(y) is the variance of the transformed signal by different transform function

Table 9 Ablation study of different inverse and different loss functions on CRVD dataset (burst number $N=5$)

Appendix B Generalized Verison of Freeman-Tukey Transformation

For For variable x in Poisson distribution of the mean value $x^{*}$, the general form of variance stabilization transformation in root-type is

$$\begin{aligned} y = 2 \sqrt{x + c}. \end{aligned}$$

(B.1)

The core problem of variance stabilization is to stabilize Poisson distribution to have unit variance. But no exact stabilization is possible Curtiss (1943). In practice, approximate transformations are generally used. The mainstreaming transformations include $2\sqrt{x}$, $2\sqrt{x+1}$, $2\sqrt{x+\frac{1}{2}}$ Bartlett (1936), $2\sqrt{x+\frac{3}{8}}$Anscombe (1948) and $\sqrt{x}+\sqrt{x+1}$Freeman and Tukey (1950). $\sqrt{x} + \sqrt{x+1}$ can be taken as the linear combination of two general forms with $c=0$ and $c=1$. We visualize the variance of transformed y in Fig. 7. When the value x is enough large, the variance of $2\sqrt{x+\frac{1}{2}}$ Bartlett (1936), $2\sqrt{x+\frac{3}{8}}$ Anscombe (1948) and $\sqrt{x}+\sqrt{x+1}$Freeman and Tukey (1950) approach the unity. However, $\sqrt{x}+\sqrt{x+1}$ Freeman and Tukey (1950) shows better approximation than other transformations when the mean value $x^{*}$ is close to zero. The SNR (signal-to-noise ratio) in dark areas is usually lower than that of other areas. Therefore, we seek the generalized version of Freeman-Tukey Transformation Freeman and Tukey (1950) to handle Poisson-Gaussian distribution for raw denoising.

Firstly, we start from the transform of Poisson distribution. We define variable x to be a Poisson variable of mean m. Its variance is $\text {Var}(x) = m$. We define y to be the transformed x. Then we have $\text {Var}(y) \approx (\frac{dy}{dx})^2\text {Var}(x)$ based on Doob (1935) and Bartlett (1947). The core problem of variance stabilization is stabilize Poisson distribution into unity variance. Hence we let $\text {Var}(y) = 1$ and obtain:

$$\begin{aligned} \frac{dy}{dx} = \sqrt{\frac{\text {Var}(y)}{\text {Var}(x)}} = \frac{1}{\sqrt{m}}. \end{aligned}$$

(B.2)

For the general transform $y = 2\sqrt{x+c}$, we have

$$\begin{aligned} \frac{dy}{dx} = \frac{1}{\sqrt{x + c}}. \end{aligned}$$

(B.3)

From Eqs. (B.2) and (B.3), we obtain the approximation:

$$\begin{aligned} m = x + c. \end{aligned}$$

(B.4)

Secondly, we consider the transform of Poisson-Gaussian distribution. Similar to Eq. (5), we define variable z as $z = x + \gamma$, where x is a Poisson variable of mean m and $\gamma$ is a Gaussian variable of mean g and standard deviation $\sigma$. The variance of transformed z is given by $\text {Var}(y) \approx (\frac{dy}{dx})^2\text {Var}(z)$ based on Doob (1935) and Bartlett (1947). Similarly, we let $\text {Var}(y) = 1$ and obtain:

$$\begin{aligned} \frac{dy}{dz} = \sqrt{\frac{\text {Var}(y)}{\text {Var}(z)}} = \frac{1}{\sqrt{m + \sigma ^2}}. \end{aligned}$$

(B.5)

We take the first-order approximation in Starck et al. (1998) to approximate the Gaussian distribution $\gamma \approx g$. From Eq. (B.4), we have $m = z + c - g$. Thus we have:

$$\begin{aligned} \frac{dy}{dx} = \frac{1}{\sqrt{z + c + \sigma ^2 - g}}. \end{aligned}$$

(B.6)

By integral of Eq. (B.6), we have the transformation y(z) for Poisson-Gaussian distribution:

$$\begin{aligned} y(x) = 2\sqrt{z + c + \sigma ^2 - g}. \end{aligned}$$

(B.7)

Finally, we move to the generalized version of Freeman-Tukey Transformation Freeman and Tukey (1950): $y = \sqrt{x} + \sqrt{x+1}$. From the Eq. (B.7), we generalize $2\sqrt{x}$ and $2\sqrt{x+1}$ respectively. By using linear combination of two generalized transformations ($c=0$ and $c=1$), we obtain the generalized version of Freeman-Tukey Transformation:

$$\begin{aligned} y(x) = \sqrt{x + 1 + \sigma ^2 - g} + \sqrt{x + \sigma ^2 - g }. \end{aligned}$$

(B.8)

Appendix C Algebraic Inverse of transform

It is known that algebraic inverse is usually avoided due to bias in previous methods Starck et al. (1998). However the bias is already handled when we calculate the loss in the space of variance stabilization. Moreover, algebraic inverse can be used for both Anscombe transformation Anscombe (1948); Starck et al. (1998) and Freeman-Tukey transformation Freeman and Tukey (1950) in our framework.

Let x and $x^{*}$ denote noisy signal and clean signal, respectively. The transform (Anscombe transform or Freeman-Tukey transform) is denoted as f and the algebraic transform is denoted as $f^{-1}$. The bias is produced by the nonlinearity of the transformation f. We calculate the loss in the variance stabilization space. The denoising network would learn the mapping from f(x) to $f(x^{*})$ directly. Therefore, the bias is already handled when the denoising output approximates $f(x^{*})$.

We further conduct experiments on CRVD dataset (burst number $N=5$) to compare algebraic inverse and exact unbiased inverse under different training settings. The results are shown in Table 9. We first training with Generalization Anscombe transformation (GAT) Starck et al. (1998) and calculate the loss function before the inverse. Then we test the model with algebraic inverse (denoted as “GAT-4”) and exact unbiased inverse (denoted as “GAT-3”). It is shown that algebraic inverse outperforms the exact unbiased inverse Makitalo and Foi (2013) by 0.13 dB PSNR, which demonstrates that the bias is handled in calculating loss before inverse. Then we train with GAT with algebraic inverse (denoted as “GAT-2”) and optimal inverse (denoted as “GAT-1”) and calculate the loss function after the inverse. In Table 9, it can be observed that both two inverses show the same performance (44.60 dB PSNR) but are 0.03 dB PSNR lower than calculating the loss before inverse. It might be because the bias produced in the space of variance stabilization becomes more complicated after the non-linear inverse transformation. Handling the bias before inverse is more direct. The same phenomenon can also observed in the Freeman-Tukey transformation (“Ours-1” VS “Ours”).

Table 10 Ablation study of different input orders of alternate frames on CRVD dataset (burst number $N=5$)

Appendix D More Ablation of Denoising Network

Input order of alternate frames We conduct experiments on CRVD dataset Yue et al. (2020) (burst number $N=5$) to compare three input orders: a) preserving the temporal order of an input burst (denoted as “Keep”), b) shuffling the burst order randomly (denoted as “Shuffle”), and c) reversing the burst order (denoted as “Reverse”). In training and testing, 4 alternate frames are re-arranged following the same ordering strategies. It is shown in Table 10 that training with preserving the temporal order achieve the best performance of 44.70 dB PSNR, which slightly outperforms random shuffling by 0.03 dB PSNR. Furthermore, reversing the temporal order achieve the worst performance of 44.63 dB PSNR, which suffers a drop of 0.07dB PSNR. It can be observed that preserving the temporal order is helpful in sequential denoising.

Specializing the network weights In our denoising network S, we have a series of sub-networks for sequential denoising. For burst denoising on CRVD dataset Yue et al. (2020) (burst number $N=5$), $S_0$ is for spatially denoising of reference frame and $S_1,S_2,S_3,S_4$ are for sequential denoising of the 4 alternate frames. We conduct experiments on CRVD dataset (burst number $N=5$) to compare $S_i$ with different weights (denoted as “specializing”) and $S_i$ with shared weights (denoted as “sharing”). It shown in Table 11 that using shared weights of $S_1,S_2,S_3,S_4$ just achieves 44.44 dB PSNR, which has a drop of 0.26 dB PSNR compared with specializing each $S_i$ (44.70 dB PSNR).

Table 11 Ablation study of using specialized or shared-weights networks on CRVD dataset (burst number $N=5$)

Table 12 Ablation study of using different numbers of frequencies in the denoising network on CRVD dataset (burst number $N=5$)

Different Scales in denoising backbone We conduct experiments on CRVD dataset (burst number $N=5$) to explore using different scales (frequencies). We define the number of scales (frequencies) as s. When $s=4$, we use four frequencies ($m_0,m_1,m_2,m_3$) to achieve multi-frequency denoising. It can be observed in Table 12 that using two frequencies achieves 44.49 dB PSNR, which is a drop of 0.21dB compared with using three frequencies (44.70 dB PSNR). But when we use four scales (frequencies), the denoising performance is 44.71 dB PSNR and just outperform using three frequencies by only 0.01 dB PSNR but its model size increases from 1.57M to 2.10M.