1 Introduction

Signals acquired by digital sensors are generally affected by different types of noise, such as speckle [1] and exponential [2] noise on biomedical images, salt-and-pepper [3], Gaussian [4], and Poisson [5] noise on images acquired through camera sensors (Sect. 2).

Our goal (Sect. 3) is the definition of a novel denoising method belonging to the low-rank class, based on the singular value decomposition (SVD) and the learning and prediction of the optimal thresholds. Given a training data set of ground-truth images (Fig. 1), we apply different types of noise (e.g. Gaussian, speckle, salt-and-pepper, Poisson, exponential noise) and extract 3D blocks through the block-matching algorithm [6]. Then, we compute the optimal singular values through a proper optimisation applied to the SVD of each 3D block. We iterate this approach, where the input image of each iteration is the denoised image at the previous step.

The input and optimal singular values compose the training data set of the learning model; the singular values of the training data set are classified according to four parameters: noise type and intensity, number of iterations of the method, and clustering of the singular values. This classification allows us to design specific networks and improve the accuracy of the learning model. The learning phase optimises a matrix of weights, applied to train the model and predict the optimal thresholds of the SVD. For the image denoising, we compute the 3D blocks and the SVD, the four parameters of each 3D block, and the thresholds through the network’s prediction. Finally, the block-matching aggregation is applied to reconstruct the denoised image (Sect. 4). As main contributions, we train our learning-based model to predict the optimal thresholds of the SVD. In contrast, learning-based methods [7] predict the denoised image; low-rank methods [8] manually select the SVD thresholds; the block-matching 3D (BM3D) [6] applies a wavelet decomposition, with a manual selection of the thresholds. Our denoising method is independent of the type of noise and provides an upper bound to the accuracy of the denoising of 2D images through the SVD. Our approach can be generalised to different transformations of the noisy signal, e.g. spectral (wavelet, shearlet), factorisation (e.g. SVD, HOSVD), or to volumetric data through spherical harmonics and Laplacian eigenpairs (Sect. 5).

2 Related work

Low-rank methods compute the denoised image as the low-rank matrix approximation of the noisy image [8]. This class of methods is extensively applied in biomedical 2D [9] and 3D [10] images, hyperspectral images [11], and 2D videos [12]. The numerical properties of the singular values thresholds are analysed [13] for low-rank matrix completion. Total variation (TV) and low-rank approximation are applied [14, 15], by iteratively updating the weight parameters of TV regularisation. The learning of an adaptive regulariser for low-rank approximation [16] assumes a Laplacian distribution of the singular values. A Wiener filter [17] is estimated by optimising a low-rank approximation, where eigenvalue thresholds are computed by using a prior non-local self-similarity. A tensor-based approach [18] applies low-rank approximation.

The weighted nuclear norm minimisation (WNNM) [19] computes the stacks as BM3D method, performs an SVD on the stacks and applies a weighted threshold to the singular values, where lower weights correspond to higher singular values. Lower singular values capture the noisy component of the image, so their reduction is larger. Finally, the collaborative filtering reconstructs the denoised image.

Spectral decomposition transforms a signal into its spectral domain and exploits the sparsity of the transformed signal to remove noise, through a threshold operation [20]. The BM3D [6] computes and stacks similar patches through the non-local means (NLM) method [21], which uses the patterns’ redundancy of the input image, and each patch is restored with a weighted average of all the other patches, where each weight is proportional to the similarity among the patches. In BM3D, each stack is transformed into its spectral domain with wavelet decomposition, denoised through a hard/soft threshold, and reconstructed in the space domain. Then, the denoised patches are aggregated by a collaborative filter to reconstruct the denoised image.

An over-complete dictionary represents the signal as a linear combination of atoms [22], which are iteratively updated through the SVD of the representation error to better fit the data. To estimate the coefficients of the sparse representation of the input image, the non-locally centralised sparse representation (NCSR) [23] exploits the non-local redundancies, combined with local sparsity properties; the dictionary is learned by clustering the patches of the image into k clusters through the k-means [24] method and then learning a sub-dictionary for each cluster.

Fig. 1
figure 1

Learning-based denoising: pipeline

Full size image
Fig. 2
figure 2

Input/noisy (1st row) and optimal denoising (2nd row) for: cameraman (\(256 \times 256\)), with speckle noise and \(\sigma =0.05\); boat (\(256 \times 256\)), salt-and-pepper noise with density 0.05; mandrill (\(256 \times 256\)) Gaussian noise with \(\mu =0,\sigma =0.01\)

Full size image
Fig. 3
figure 3

1st Row: PSNR (\(y-\)axis) comparison of the number of iterations (\(x-\)axis) between optimal (Sect. 3.1) values (blue), and our method’s values (red). 2nd Row: average optimal thresholds (\(y-\)axis) of each singular value (\(x-\)axis). a Cameraman (\(256 \times 256\)), speckle noise (\(\sigma =0.05)\); b boat (\(256 \times 256\)), salt-and-pepper noise (density 0.05); c mandrill, Gaussian noise (\(\mu =0,\sigma =0.01\)). Results refer to Fig. 4 (color figure online)

Full size image

3 Learning low-rank denoising

An input image \(\mathbf {Y}=f(\mathbf {X},\mathbf {N})\) is composed of the ground-truth \(\mathbf {X}\) and the noisy component \(\mathbf {N}\), where f defines the combination of one or more types of noise (e.g. additive, impulsive, multiplicative). Low-rank approximations recover the underlying low-rank matrix from its degraded observation. In this class, the nuclear norm minimisation recovers an estimation \(\hat{\mathbf {X}}\) of the ground-truth signal through the minimisation of the energy functional \(\Vert \mathbf {Y} - \mathbf {X} \Vert _{F}^{2}+\lambda \Vert \mathbf {X} \Vert _{*}\), where the nuclear norm \(\Vert \mathbf {X} \Vert _{*}\) is the sum of the singular values of \(\mathbf {X}\), and measures the compliance of the approximated image to the high-frequency components of the input image. This problem is equivalent to recover the approximated image \(\hat{\mathbf {X}} = \mathbf {U}(\tau (\mathbf {S}))\mathbf {V}^{\intercal }\), where \(\mathbf {Y} = \mathbf {U}\mathbf {S}\mathbf {V}^{\intercal }\), \(\mathbf {S}:=\text {diag}(\mathbf {s})\), is the SVD of the noisy signal, and \(\tau (\mathbf {S})=\text {diag}(\mathbf {s}-{\lambda })\) with \({\lambda }\) constant vector. The thresholding \(\tau (\mathbf {S})\) improves the image reconstruction by removing the noise component while preserving image features. Weighted thresholds imply a different shrinkage to each singular value, as \(\mathbf {S}_{hh} - {\lambda }_{h}\), where \(\mathbf {S}_{hh}\) is the (h, h) entry of \(\mathbf {S}\). The proposed learning-based method trains a network to predict the optimal thresholds \({\lambda }_{h}\). We present the computation of the optimal thresholds (Sect. 3.1), the related denoising accuracy (Sect. 3.2), the setting (Sect. 3.3), and optimisation (Sect. 3.4) of the learning model for the prediction of the thresholds.

Fig. 4
figure 4

Comparison between previous work and our denoising: a cameraman (\(256 \times 256\)), speckle noise with \(\sigma =0.05\); b boat (\(256 \times 256\)), salt-and-pepper noise with density 0.05; c mandrill, Gaussian noise with \(\mu =0,\sigma =0.01\); d magnification

Full size image

3.1 Optimal thresholds

Given a \(n \times n\) ground truth image \(\mathbf {X}\) and a noisy instance \(\mathbf {Y} = f(\mathbf {X},\mathbf {N})\), we compute the optimal thresholds \({\lambda }_{h}\) through the minimisation of the distance between the ground-truth and the reconstructed signal \(\hat{\mathbf {X}}\), as

$$\begin{aligned} {\small { E({\lambda }) = \sum _{i=1}^{n}\sum _{j=1}^{n}\vert \mathbf {X}_{ij} - \hat{\mathbf {X}}_{ij} \vert ^{2},}} \end{aligned}$$
(1)

where the reconstructed signal is computed through the SVD of the \(\mathbf {Y}\) signal with weighted thresholds, i.e. \(\hat{\mathbf {X}}_{ij} = \sum _{h=1}^{n}\mathbf {U}_{ih}(\mathbf {S}_{hh}-{\lambda }_{h})\mathbf {V}_{hj}\). The first-order derivatives of the energy functional \(E({\lambda })\) with respect to the variables of the optimisation problem \({\lambda }\) are \(\partial _{\lambda _{k}} E = 2\sum _{i,j=1}^{n}\left[ \mathbf {X}_{ij} - \sum _{h=1}^{n} \mathbf {U}_{ih}(\mathbf {S}_{hh}-{\lambda }_{h})\mathbf {V}_{hj}\right] \mathbf {U}_{ik}\mathbf {V}_{kj}\).

The minimum of Eq. (1) is computed through L-BFGS (Limited-memory Broyden, Fletcher, Goldfarb, Shanno) [25], which finds the roots of the derivative of the energy functional. L-BFGS is an iterative optimisation algorithm in the family of quasi-Newton methods that estimates the inverse Hessian matrix for the minimum search in the variable space, by representing the approximation as a few vectors. At each iteration, a small history of the past updates of the variables \(({\lambda })\) and of the gradient of the energy functional \(E(\cdot )\) in Eq. (1) is applied to identify the direction of steepest descent and to implicitly perform operations requiring vector products with the inverse Hessian matrix. The memory storage is \(\mathcal {O}(u^{2})\) and the computational cost is \(\mathcal {O}(u v)\) at each iteration, where u is the total number of variables, and v is the number of steps stored in memory.

3.2 Optimal denoising thresholds

Through the optimal singular values thresholding, we set an upper limit to the accuracy of denoising.

Application of block matching Given a pixel y, the patch \(P_{y}\) is the set of pixels in the neighbourhood of y. The search window is the set of selected patches used to search the closest ones to a reference patch, according to certain metrics. The stack, or 3D block, is the set of patches similar to a reference patch. The block-matching algorithm stacks the similar patches into a 3D structure and uses the redundancy of the stack to remove the noise. Then, the patches are aggregated by a collaborative filter to compose the denoised image.

Optimal denoising: results We compute the optimal denoising of three images with a different noise. The method is applied for 20 iterations: at each iteration, the denoised image of the previous iteration is used as a noisy image, while the ground truth is fixed. We compute the 3D blocks with the block-matching algorithm and extract their ground-truth counterparts from the ground-truth image. Then, we compute the optimal thresholds \({\lambda }_{h}\) for the SVD through the minimisation of Eq. (1) for each 3D block, and we reconstruct the denoised image.

Considering the Gaussian, speckle, and salt-and-pepper noise (Fig. 2), the denoised images after 20 iterations are very close to the input image, thus showing that the nuclear norm minimisation recovers the ground-truth image with any type of noise, when the proper thresholds are selected. Most of the details are well reconstructed, such as the cameraman’s face, the boat profile, or the mandrill zygoma. Images with a different noise, in terms of both type and intensity, have a different reconstruction accuracy. This result motivates us to distinguish learning-based networks according to the type of noise (Sect. 3.3). The optimal PSNR plot (Fig. 3, 1st row, blue line) provides the upper bound of the optimal denoising and allows us to compare the accuracy of our learning-based method with the optimal achievable results.

Optimal denoising: thresholds In Fig. 3(2nd row), the averaged optimal thresholds at the first iteration: \(\bar{{\lambda }}_{h}^{it=1}=\sum _{j=1}^{B}{\lambda }_{hj}^{it=1}/B\), where B is the total number of 3D blocks. Different types of noise have similar behaviour; for example, in (a) the first singular value is reduced to an average value of 0.14, which is less than the \(1\%\) of the average input value. Then, the optimal thresholds increase up to 1.2 for the fourth singular value, which is more than the \(73\%\) of the input singular value. Finally, the thresholds decrease: The reduction of the 49th singular value is about 0.2.

3.3 Training and clustering for learning

Applying an arbitrary noise to a set of input images, we build a data set of ground truth and noisy images to train a learning-based method to predict the optimal thresholds. In particular, the input of the network is the vector of the singular values of the noisy block and the target is the vector of the optimal thresholds (Eq. (1)).

Instead of defining a single data set for all the images, we organise the images according to four parameters: (i) type of noise, (ii) iteration, (iii) noise intensity, and (iv) singular values clustering. This classification improves the specificity of the network and the accuracy of the prediction of the thresholds. The four parameters are nested: For each image noise (i) and iteration step (ii), we compute the 3D block’s noise intensity (iii) with the nonlinear noise estimator (NOLSE) [26] metric and define c intervals so that the NOLSE values are uniformly distributed into c classes. We cluster the singular values (iv) into k groups with the k-means algorithm; then, we train a support vector machine (SVM) [27] to classify the vectors of singular values according to the k-means labels. For each combination of the four parameters, we define the matrix \(\mathbf {P} = \{\mathbf {P}_{k}\}_{k=1}^{N}\), with \(\mathbf {P}_{k} = [p_{1},\dots ,p_{n}]^{\top }\), where N is the number of 3D blocks and n is the lower dimension of the 3D block; the p elements are the singular values computed through the SVD. These values are associated with the corresponding optimal thresholds \(\mathbf {Q} = \{\mathbf {Q}_{k}\}_{k=1}^{N}\), computed as the minimisation of Eq. (1).

3.4 Learning model

For each combination of the four parameters (Sect. 3.3), we compute a dedicated learning-based network. Given the singular values of a 3D block, we train a network to predict the thresholds that reconstruct the best approximation of the ground-truth block. The network weights are defined through the matrix \(\mathbf {W}\) of dimension \(n \times n\), and are computed by minimising the distance between the target and the predicted thresholds, as \(F(\mathbf {W}) = \sum _{k=1}^{N} \sum _{j=1}^{n} \left| \mathbf {Q}_{jk} - \hat{\mathbf {Q}}_{jk} \right| ^{2}\), where \(\hat{\mathbf {Q}}_{jk} = \sum _{i=1}^{n} \mathbf {W}_{ij}\mathbf {P}_{ik}\) are the predicted thresholds. The properties of the functions are discussed in the additional material.

Fig. 5
figure 5

PSNR (\(y-\)axis) average values of our and state-of-the-art methods (\(x-\)axis) with Gaussian (a), speckle (b), salt-and-pepper (c) noise

Full size image

4 Experimental results

We present the network training (Sect. 4.1), our denoising (Sect. 4.2), and thresholds (Sect. 4.3).

Fig. 6
figure 6

a Partition of the elements (i.e. input and target vectors) for the training. Each noise intensity class (\(x-\)axis) is divided in 3 k-means classes (blue, orange, and yellow); \(y-\)axis shows the cumulated number of elements of each class. b Input (blue), optimal (orange), and predicted (yellow) singular values (\(y-\)axis) of the cameraman image with speckle noise, for the 49 (\(x-\)axis) singular values. c Prediction error (\(y-\)axis) of the 49 singular values (\(x-\)axis) of a 3D block of the cameraman image with speckle noise. d Partition of the prediction error (\(y-\)axis), when varying the noise intensity classes (\(x-\)axis): each class is composed of three SVM-predicted classes (i.e. blue, orange, and yellow) (color figure online)

Full size image

4.1 Training results

We select a data set of 39 images [28], of different resolutions and subjects. We apply an artificial noise to the images (e.g. Gaussian noise with mean 0 and variance 0.01); then, we compute the 3D blocks with block matching. We compute the input singular values and the target thresholds (Sect. 3.1). Images are classified according to the noise type (e.g. speckle, Gaussian) and intensity (e.g. \(\sigma = 0.05\) in speckle noise) of the input image. Then, we split the data into \(c=5\) 3D block noise intensity classes, and \(k=3\) k-means labels (Sect. 3.3), for a total of 15 instances for each iteration. The final data set is composed of a set of 15 matrices \(n \times N\), where N is the total number of 3D blocks, and n is the number of singular values (i.e. 49 in our tests, corresponding to a \(7 \times 7\) patch). This approach is repeated at each iteration: for the computation of the data set at iteration t, the block matching is applied for \((t-1)\) iterations. At each iteration, the proper networks previously computed by the minimisation model (Sect. 3.4) are used to predict the thresholds and denoise the related 3D block. At iteration t, the optimal thresholds are computed through an optimisation step (Eq. (1)). In our tests, we apply 5 iterations of the training, for a total number of 75 networks (i.e. \(5 \times 3 \times 5\)) for each type of noise and noise intensity.

Computation time The computation of the input/optimal thresholds (Eq. (1)) and of the network weights (Sect. 3.4) are performed offline. For the computation of the training data set, we used 39 images, for a total number of 257,240 couples of input/target vectors for each noise type and iteration. The execution time for the generation of the data set of each noise type/intensity is approximately 120 minutes for each iteration. For the computation of the weights, the variables of each network are \(49 \times 49\) matrices; we consider 3 types of noise (i.e. salt and pepper, speckle, Gaussian), 5 iterations for each noise, 5 3D block’s noise intensity classes, and three \(k-means\) labels, for a total number of 225 networks. The execution time of the training of each network is about 15 minutes. Tests have been executed with MATLAB R2020a, on a workstation with 2 Intel i9-9900KF CPUs (3.60 GHz), and 32GB RAM. Additional tests with increased noise intensity (i.e. Gaussian and speckle) and different types of noise (i.e. Poisson and exponential) are discussed in the additional material.

4.2 Denoising results

We discuss the generality of our method from the type of noise applied. Applying a speckle noise with \(\sigma =0.05\) (Fig. 4a), our method improves previous work and shows better preservation of the main features and details (e.g. the face of the man, the folds of the man’s trousers), and good noise removal (e.g. the sky in the background). Furthermore, our method does not introduce blurred patterns. In comparison, WNNM removes a larger amount of noise, while losing several details (e.g. man’s mouth), and increasing the blurring effect. On the boat image with salt-and-pepper noise with 0.05 density (Fig. 4b), our method better preserves main details, such as the clouds, or the geometries of the boat. In comparison, BM3D does not remove the “salt” noise on the bottom side of the boat. On the mandrill image with \(\mu =0\) and \(\sigma ^2=0.01\) Gaussian noise (Fig. 4c), all methods have similar results, correctly remove the noise and preserve the main features of the image (e.g. the nose and the eyes of the mandrill). Only the mandrill’s pupil in the NCSR method is less uniform than our method.

Table 1 summarises the quantitative metrics of our method and previous work. With Gaussian noise, our method has comparable results with state-of-the-art methods, which are specific for this type of noise. In (c), the PSNR value is 26.01 for our method, 26.13 for WNNM (best result), and lower than 26 for BM3D and NCSR. Instead, our method has better results when other types of noise (e.g. salt and pepper, speckle) are applied. In Fig. 3, we compare the PSNR of our method with the optimal PSNR results (Sect. 3.1); in (a), our method has a PSNR value of 28.69 after 4 iterations, while the optimal PSNR value is 30.53. The error of the network prediction of the thresholds, even if small, leads our method to diverge from the optimal reconstruction. After five/six iterations, our method is not able to further improve the image approximation. Considering a large data set [29] of natural images of different resolutions and subjects, and applying different types of artificial noise, our method has slightly better results than previous work in the case of Gaussian noise, while it widely improves the PSNR values in the case of speckle and salt-and-pepper noise (Fig. 5).

Computation time The execution time for denoising each 3D block is 0.015 seconds; this time is composed of the noise intensity computation through the NOLSE metric (\(66\%\)), the SVD (\(5\%\)), the SVM prediction of the label of the singular values (\(14\%\)), and the network prediction of the optimal thresholds (\(15\%\)). Due to the operations performed for each 3D block, our method (330 s) increases the execution time for the denoising of a \(256 \times 256\) image, with respect to previous work: BM3D (1 sec.), WNNM (100 s), and NCSR (180 s).

Table 1 PSNR and SSIM metrics of our denoising, compared with previous work: (a) speckle, (b) salt-and-pepper, and (c) Gaussian noise
Full size table

4.3 Selected thresholds

The training data set of the Gaussian noise with \(\mu =0\) and \(\sigma ^2=0.01\) is composed of more than 257K elements, each one composed of a couple of vectors (i.e. input and optimal target) in \(\mathbb {R}^{49}\). Figure 6a shows the partition of these elements, according to the 3D block noise intensity and the k-means classification. Since each 3D block noise intensity class contains the same number of elements, we have a regular partition of the training data set. In contrast, the k-means classification does not provide a uniform partition of the elements; indeed, some networks are trained with more data than others. However, each network has a sufficient number of elements due to the huge data set; in fact, the minimum number of elements is \(N=4204\), related to 3D block noise class \(c=1\) and k-means label \(k=3\).

Figure 6b shows a comparison among the input, optimal, and predicted singular values of a specific 3D block. The optimal singular values reflect the thresholds in Fig. 3(2nd col.); for example, the shrinkage of the second singular value is larger than the first one. The predicted singular values are very close to the optimal singular values; in fact, the error for the prediction of a \(49-\)dimensional vector is 0.6, which means an average error of 0.01 for each element of the vector. Analysing the error between the prediction and the optimal singular values \((\mathbf {Q}_{jk} - \hat{\mathbf {Q}}_{jk})_{j=1}^{49}\) (Sect. 3.4), the first values show a higher prediction error than the last ones (Fig. 6c), due to the different magnitude and variability of these values. However, there is not a clear correlation between these factors. The correlation between the prediction error of the thresholds and the data clustering (Sect. 3.3) shows that the error increases with the 3D block noise intensity (Fig. 6d), as the noise intensity implies a more irregular distribution of the singular values, which is harder to train and predict. Instead, the SVM-predicted classes and the prediction error of the network do not show any correlation.

5 Conclusions and future work

We have proposed a denoising method that learns the optimal thresholds of the SVD from a training data set and applies a prediction of these optimal values to improve the image reconstruction. As main contributions, the proposed method improves previous work on denoising, and it is general with respect to the noise type and intensity. We have also analysed the upper bounds of the nuclear norm minimisation for the denoising of 2D images, and the learning results for the prediction of the optimal thresholds of the SVD. As future work, we plan to improve the approximation accuracy of the ground-truth image, by reducing the error of the thresholds prediction, extending the proposed method to different data (e.g. 3D images, 2D videos), and signal decompositions (e.g. wavelets). Trained networks are available at https://github.com/cammarasana123/denoise