Poisson noise removal from high-resolution STEM images based on periodic block matching
- First Online:
- Received:
- Accepted:
- 8 Citations
- 2.4k Downloads
Abstract
Scanning transmission electron microscopy (STEM) provides sub-ångstrom, atomic resolution images of crystalline structures. However, in many applications, the ability to extract information such as atom positions, from such electron micrographs, is severely obstructed by low signal-to-noise ratios of the acquired images resulting from necessary limitations to the electron dose. We present a denoising strategy tailored to the special features of atomic-resolution electron micrographs of crystals limited by Poisson noise based on the block-matching and 3D-filtering (BM3D) algorithm by Dabov et al. We also present an economized block-matching strategy that exploits the periodic structure of the observed crystals. On simulated single-shot STEM images of inorganic materials, with incident electron doses below 4 C/cm ^{2}, our new method achieves precisions of 7 to 15 pm and an increase in peak signal-to-noise ratio (PSNR) of 15 to 20 dB compared to noisy images and 2 to 4 dB compared to images denoised with the original BM3D.
Keywords
Image denoising Poisson noise Non-local means Block matching 3D transform shrinkage Periodic search STEM PrecisionAbbreviations
- STEM
scanning transmission electron microscopy
- BM3D
block-matching and 3D-filtering algorithm
- PSNR
peak signal-to-noise ratio
- SNR
signal-to-noise ratio
- NLM
non-local means
- AGWN
additive Gaussian white noise
- HRTEM
high-resolution transmission electron microscopy
- VST
variance-stabilizing transform
Background
Modern electron microscopy allows for atomic resolution images of crystalline structures [1]. However, the full scope of resolution can be exploited only for materials with little electron beam sensitivity. Lowering the electron dose decreases the signal-to-noise ratio (SNR) of the acquired micrographs accordingly, degrading the quality or even completely prohibiting the extraction of desired information from the noisy micrographs. Examples of inorganic materials with high beam sensitivity, where scanning transmission electron microscopy (STEM) images of poor SNR have to be used, are both oxide [2] and metallic [3] catalysts. One important quantity that may be extracted from atomic-resolution electron micrographs is the positions of the atoms. The precision with which these can be determined is crucial for the understanding of certain material properties [4,5]. For single-shot STEM images, the best reported precision is about 15 pm [6]. A common technique to increase the precision is to register a series of frames and extract the atom positions from the average of the registered frames. Kimoto et al. [7] achieved a precision of 5 pm using rigid registration. More recently, a non-rigid registration scheme has been developed by Berkels et al. [2] that has achieved sub-picometer precision for STEM series [8]. Both registration schemes require many individual frames of moderate SNR resulting in a very-high overall electron dose applied to the material. Thus, to widen the applicability of STEM to more beam-sensitive inorganic materials, a central objective is to develop effective denoising methods that increase the single-shot image quality. This would enable the extraction of desired information from individual frames or the use of the individual frames for registration, while using a lower electron dose.
Denoising in materials, electron microscopy is often accomplished by simple spatial filters like a Gaussian blur or median filter or frequency space low-pass or Wiener filters (cf. [9-11]). These examples are all taken from materials robust enough under the electron beam to survive relatively high electron dose to obtain images with high spatial oversampling (small pixels). More sophisticated approaches that take advantage of the highly redundant nature of some high-resolution STEM images have been employed [12], building on tools developed for very beam-sensitive biological samples [13-16]. While these methods, mostly based on manipulations of the image in Fourier space, are well established and have proven to perform well on high-resolution transmission electron microscopy (HRTEM) images of organic materials, they are tightly linked to the periodicity of the data and have thus not been widely adopted in STEM studies of aperiodic defects in inorganic materials.
The most successful generic image denoising method for arbitrary images available today rely on non-local detection and averaging of self-similar image regions. The first algorithm based on this strategy is the non-local means filter (NLM) by Buades et al. [17]. Due to the richness in self-similarity of electron micrographs of crystals, NLM is in principle very well suited for denoising such micrographs [18], and it has been employed in biological electron microscopy [19]. However, two main properties of low-dose electron micrographs of crystals are disregarded by the original design and implementation of NLM, namely (1) NLM removes additive Gaussian white noise (AGWN) instead of Poisson noise, which is the dominant form of noise in low-dose STEM images; and (2) NLM uses a local similarity search to reduce the computational cost, whereas the distance between self-similar points in images of crystals is at least as large as the inter-atomic distance. In the case of STEM, there is an additional difficulty caused by image distortions resulting from the serial acquisition of the pixel data. In view of these issues, we proposed an enhanced version of NLM tailored to STEM imaging [20]. The most significant enhancement was the development of an efficient non-local similarity search, based on the generation of periodic lattices in Fourier space [21]. However, the basic NLM principle can be further improved by replacing the simple weighted average of intensities of pixels with similar neighborhoods by a more advanced collaborative filtering of the corresponding image parts. Several methods of this type have been proposed over the past few years, e.g., optimized block-wise NLM by Coupé et al. [22] and NLPCA by Salmon et al. [23], which is based on principal component analysis. Among the most successful variants is the block-matching and 3D-filtering algorithm (BM3D) developed by Dabov et al. [24], which has become a benchmark for image denoising algorithms in the field of image processing. The extension of BM3D to Poisson noise removal via application of the Anscombe variance-stabilizing transformation [25] has already been proposed by Mäkitalo and Foi [26] who have contributed an exact unbiased inverse transformation that increases the accuracy especially in the low-count regime. In [20], it was found that despite being restricted to a local similarity search, the original BM3D filter with extension to Poisson noise outperformed the proposed periodic search NLM filter. Thus, the starting point here is the BM3D filter.
In this paper, we discuss how modifications introduced in STEM-tailored NLM [20] can be incorporated into the state-of-the-art denoising algorithm BM3D in order to tap its full potential on electron micrographs of crystals. More precisely, as this method has been designed for Gaussian white noise, a central task is to develop suitable modifications that effectively deal with Poisson noise. Another focus is to exploit the atomic lattice structure that entails a repeated appearance of self-similar image components within a single frame. While we focus on the application of STEM imaging of inorganic materials, the key features of Poisson noise and oversampling are shared by HRTEM images as well, so the proposed method should also be applicable to HRTEM images. Also note that the procedures proposed in this paper can be applied subsequently for registering a series of low-dose frames to a single reference frame [2].
The paper is organized as follows. First, we briefly recall the original BM3D algorithm. Then, we present two strategies for Poisson noise removal, namely variance stabilization based on the Anscombe transform and using Poisson maximum-likelihood similarity measures due to Deledalle et al. [27] for the block matching. The main contribution is an adaptive non-local periodic block matching. It exploits the repetitive structure within the micrograph while keeping the computational cost affordable. Since the spatial distribution of the resulting block estimates is highly non-uniform, potentially resulting in a poor reconstruction in regions with few available block estimates, we discuss possible remedies of this deficiency. Finally, we evaluate the proposed method on simulated data, where the ground truth, i.e., the true, noise-free mean electron count per pixel, is available, showing the improvement both in terms of visual image quality and in quantitative measures such as peak signal-to-noise ratio (PSNR), precision, and fidelity achieved by the proposed method.
Methods
Block matching and 3D filtering
BM3D was developed on the basis of NLM. It was originally designed to remove additive Gaussian white noise from natural images or other images exhibiting a sufficient amount of self-similarity. The main idea of the algorithm is to find similar parts within an image and remove noise from those parts by sparsifying their common representation in a 3D transform domain. After processing all image parts in this manner, one receives an overcomplete representation of an estimate of the noise-free input image. Averaging all partial estimates in overlapping regions provides an estimate of the full underlying ground truth image. The denoising process in the 3D transform domain is performed in two stages, namely an initial hard thresholding succeeded by a final Wiener filtering. In the following, we briefly recall the definition of the original BM3D filter as published in [28], which serves as a basis for the methods developed in this work. We use a slightly simplified version of the algorithm described in [28] but with the exception of a change of the block size from 8×8 to 16×16, all relevant parameters, as well as the unitary transforms in the 3D domain, are chosen in exact agreement with the original implementation of the BM3D filter [29] when using the default profile, i.e., normal profile.
The original BM3D algorithm for Gaussian noise removal
where f is the (noise-free) ground truth and \( \eta \left(\cdotp \right)\sim \mathcal{N}\left(0,{\sigma}^2\right) \) is AGWN, i.e., a normally distributed random variable with zero mean and standard deviation σ.
for the tensor \( {\mathcal{Z}}_x \). A typical transformation would be based, e.g., on a wavelet basis. To this estimator, one assigns a weight \( {w}_x={N}_{\theta}^{-1} \), where N_{θ} is the number of retained non-zero transform coefficients after hard thresholding.
where χ_{Z} denotes the characteristic function of the block \( Z\subset X \). Note that each block Z_{y} may be matched to different references \( {Z}_x,{Z}_{x^{\prime }} \), resulting in different estimates \( {\hat{Y}}_{x,k},{\hat{Y}}_{x^{\prime },{k}^{\prime }} \) for the same block, i.e., \( {\left({\hat{S}}_x\right)}_k={\left({\hat{S}}_{x^{\prime }}\right)}_{k^{\prime }}=y \) in this case.
The final estimate \( \widehat{f} \) of the desired image f is now calculated in a second iteration, where the whole procedure described above is repeated with some modifications briefly indicated next, referring to [28] for the details. Denoting by F_{x}, the n×n block with upper-left corner x extracted now from the basic estimate \( {\widehat{f}}^{\mathrm{basic}} \), the above procedure is applied to the blocks F_{x} in place of Z_{x}, where the analogs to the sets S_{x} in (3) are formed, however, with respect to a possibly different threshold parameter τ^{′}. Moreover, the analog to (6) may involve a different unitary transformation (e.g., using local cosine transforms) and, perhaps more importantly, hard thresholding is replaced by Wiener filtering. Again, we refer to [28] for the details.
Figure 1 illustrates the BM3D procedure using a STEM image as an example. It demonstrates the three main steps of BM3D, which are performed for each reference block: 1) matching blocks to a reference and stacking the matched blocks into a 3D tensor, 2) denoising this column of matched blocks, and 3) aggregating the resulting denoised blocks at their original positions and averaging overlapping parts of adjacent blocks (green). For simplicity, the illustration uses simple averaging along the column of matched blocks (black dotted arrow) as an example procedure for denoising the 3D tensors. In practice, the approaches described above provide better performance.
Like any block-averaging or non-local means approach, our method has reduced performance near the edges of the image. Since N_{step}<n for every y∈X, there is a block with corner x∈X_{R} such that y∈Z_{x}. In other words, for any pixel in the image (even at the boundary), there is a block containing this pixel, resulting in an estimated intensity of the pixel after the corresponding block has been denoised in 3D transform domain. But because there exist fewer overlapping blocks near the image boundaries, there are fewer available estimates of boundary pixels that can be used for further averaging. Therefore, both the basic and final estimates \( {\widehat{f}}^{\mathrm{basic}} \) and \( \widehat{f} \), i.e., the intermediate result and the final denoised image, will show a slightly reduced quality towards the image boundaries. However, the denoising of each block together with its matched blocks in 3D transform domain contributes much more to the image quality than the averaging of spatially overlapping blocks. Thus, the reduced quality at the boundaries of images denoised with BM3D is usually not substantial in practice.
Extension to Poisson noise removal via variance stabilization
Here, given random variables V,W, \( \mathbb{E}\left\{V\Big|W\right\} \) denote the expectation of V given W. In their implementation [30], Mäkitalo and Foi provide tabulated values of this inverse for arguments within the range [ A(0),100]. For smaller arguments, the inverse is extended by zero, and for higher arguments, the asymptotic unbiased inverse transform is used, see [25].
The procedure for Poisson noise removal is then given by the following three-step procedure: first, apply the forward Anscombe transform on the input image v, receiving an input image z=A(v) with a noise model similar to AGWN; second, apply the BM3D filter to this image z, receiving an estimate \( \widehat{f} \) of E{A(v)|λ}; and lastly, apply the exact unbiased inverse Anscombe transform on \( \widehat{f} \) to obtain an estimate \( \widehat{\lambda}:={A}^{-1}\left(\widehat{f}\right) \) of λ.
Direct block matching for Poisson noise statistics
As pointed out by Salmon et al., the Anscombe transform is not accurate in the extreme low-count regime, i.e., if the average number of counts λ(x) is smaller than three for some x∈X (cf. [23]). STEM images with such a small number of counts are generally deemed useless, so they are rarely acquired or published. However, the ability to extract information from extreme low-dose images would be a substantial advantage for a variety of problems, such as characterization of metallic catalyst particles [3] and polymers and molecular crystals [31].
The performance of two variants of non-local means on images affected by Poisson noise are compared in [20]. The first one features the Anscombe transform and the standard L^{2}-patch similarity measure, while the second one uses a Poisson maximum-likelihood-based patch similarity measure from [27] without a preceding VST on the input data. The results show that when the PSNR of the input image is below 10 dB, the Poisson maximum-likelihood ratio-based patch distances outperform the strategy of using standard L^{2}-distances on the Anscombe transformed data.
we have that 0≤pml(k_{1},k_{2})≤1 for any \( {k}_1,{k}_2\in \mathbb{N} \). From this, it follows that also \( \overline{\mathrm{pml}}\left({Z}_x,{Z}_y\right) \), the geometric mean of the maximum-likelihood ratios between the blocks Z_{x} and Z_{y} takes values between 0 (very bad match) and 1 (perfect match). Thus, the corresponding threshold τ^{P} should be chosen between 0 (no thresholding) and 1 (only accepting identical blocks). We found that the choice τ^{P}=0.55 works well in practice and thus we use this value wherever the Poisson maximum-likelihood ratios are employed in this work. The reduced set \( {\check{{S}_{x}^{P}}} \) and the block stack \( {\mathcal{Z}}_x^P \) are defined analogously to \( {\check{S}}_x \) and \( {\mathcal{Z}}_x \), respectively, just replacing S_{x} by \( {S}_x^P \).
In other words, when using this strategy, the Anscombe transformed data is used to fill the 3D block stacks with the values of the matched blocks but not to determine the positions of the matched blocks in the 3D stacks.
Here, \( {\hat{Y}}_{x,k}^P(y) \) is defined in analogy to (8). Also note that due to the application of the Anscombe transform to the 3D block stacks in (19), the values of the basic estimate (20) after Poisson maximum-likelihood-ratio-based block matching are still in the Anscombe transform domain. Thus, the Wiener filtering iteration remains unchanged.
Adaptive non-local periodic block matching
Here, α_{1},α_{2} denotethe angles or directions of the periodic pattern within the input image and Δx_{1},Δx_{2} the spacings between the self-similar objects along those axes and \( {N}_S^{\pi } \) is the size of the small local search windows (in units of pixels). For an automatic estimation strategy of these parameters from the input image, we refer to [20]. Note that the \( \arg \min \) expression causes an adaptive reset of the search pattern that gives some robustness against errors in both the estimation of the grid parameters and slight variations of the periodic pattern within the image.
Here, the super scripts ht and wie refer to the first and second stage involving hard thresholding and Wiener filtering. In this paper, we have used small \( {N}_S^{\pi}\times {N}_S^{\pi } \) local search windows of size \( {N}_S^{\pi }=5 \). We found that this choice retains reasonable computational efficiency, while being large enough (about 0.1 to 0.25 times as small as the atoms in the images we used) to correct the positions of the non-local periodic search steps within each atom. \( {\check{\varPi}}_x^{\mathrm{ht}} \) and \( {\check{\varPi}}_x^{\mathrm{wie}} \) are defined analogously to \( {\check{S}}_x \). Note that for crystals at atomic scale the self-similarity within the corresponding image is so rich that with a non-local search like this, for nearly all x∈X_{R}, one finds many more than N_{3D} blocks that are similar to the reference at x, i.e., N_{x}=N_{3D} holds for nearly all x∈X_{R}.
In order to reduce the computational cost of the non-local periodic search, we made the following technical adjustments to its implementation in comparison with its description in [20] and the expression (21). First, for each reference point x among the two directions α_{1} and α_{2}, the corresponding axis with the largest intersection with the image X is declared to be the primary search axis. Then, after performing one step along the primary search axis, non-local periodic search steps along both positive and negative directions of the secondary search axis are carried out only until the image boundary is reached. This process is repeated until the image boundary is also reached along the positive and negative direction of the primary search axis. On the one hand, this implementation requires less computational cost than computing the whole set of points within the image that could be reached by steps along either of the two periodicity axes, while on the other hand, it still gives a sufficiently large subset of all of these points. This efficient periodic block-matching strategy increases the computational cost by a factor of about 1.5 to 2 (depending on the density of the atoms within the image) compared to our implementation of BM3D with local block matching. Note that due to the overhead produced by unoptimized parts of our implementation this relative comparison may not be exact. Nevertheless, we believe that the factor between the run-times of local and periodic block matching would be of the same order of magnitude in optimized code as well.
Uniform distribution of block estimates
where \( \mathcal{P}:=\left\{{\left({\varPi}_x^{\prime}\right)}_{x\in {X}_R}:{\varPi}_x^{\prime}\subset {\varPi}_x^{\ast },\#\left({\varPi}_x^{\prime}\right)={N}_x,\kern0.60em \forall x\in {X}_R\right\} \) denotes the feasible set. Unfortunately, this optimization problem is of combinatorial type and due to its global coupling of all reference coordinates would be computationally too costly to solve. Hence, we propose the following simplification of the uniform block matching through an iterative procedure.
Here, the reference coordinates have been numbered \( {X}_R=\left({x}^1,\dots, {x}^{\#\left({X}_R\right)}\right) \) and block matching is performed for one after the other in an iterative fashion. In this paper, we use the number of local \( {N}_S^{\pi}\times {N}_S^{\pi } \) search windows as the limit \( {N}_{3\mathrm{D}}^{\ast } \). Note that as the number of search window is dependent on the reference position x∈X_{R}, so is the new limit \( {N}_{3\mathrm{D}}^{\ast }={N}_{3\mathrm{D}}^{\ast }(x) \), unlike the limit N_{3D} chosen before, which is independent of x. This choice can be motivated as follows: if the adaptive periodic search succeeds in placing the local search windows roughly according to the pattern of the observed crystal, then for a reference pixel within an atom, each local search window within the corresponding similarity search should overlap a similar part of another atom. Therefore, we expect to find at least one well-matching block per local search window. Note that we define \( {n}_{\mathrm{aggr}}^U \) in analogy to (23) but with \( {\check{\varPi}}_x \) replaced by \( {\check{\varPi}}_x^U \).
Results and discussion
np- ⋆: normal profile and Anscombe-based three-step procedure
pml- ⋆: Poisson maximum-likelihood ratio-based block matching within hard thresholding iteration
⋆-l ⋆: local block matching with N_{S}=39
- ⋆- π⋆: adaptive periodic block matching with \( {N}_S^{\pi }=5 \)
⋆- πN⋆: selection of N_{3D} best-matched blocks with thresholds N_{3D}=16 (HT) and N_{3D}=32 (Wiener)
⋆- πU⋆: uniformly distributed selection of N_{3D} blocks from \( {\widehat{N}}_{3\mathrm{D}} \) best-matched blocks as in Algorithm 1
⋆- ⋆n: block size is n×n
This quantitative measure requires the knowledge of the underlying ground truth, i.e., the exact average counts λ(x) in each pixel x∈X, which is not available in experimental STEM images. Therefore, we have simulated STEM images of various materials using the frozen phonon multislice algorithm [32] to obtain representative ground truth micrographs that are free of Poisson noise (cf. Figure 2A,C,E,G). To make these images representative of experimental images, typical image distortions that are caused by instabilities of the sample and electron beam during experimental STEM image acquisition have been artificially introduced to these simulated images. Known material crystallographic data was used to create the gallium nitride and silicon atomic models. Molecular dynamics was used to calculate the silicon dislocation atomic model [8]. In order to compare the algorithms at different noise levels, we scaled the intensities of the ground truth images to different average electron counts per pixel before applying random Poisson noise. This simulates the usage of different beam currents, resulting in different electron doses. Examples of Poisson noise affected versions of the simulated images can be found in Figure 3B,D,F,H.
In this work, we use our implementation of BM3D with local search as the ‘original’ BM3D benchmark. This comparison is justified, since we verified our implementation against the implementation provided by the original authors [29] and found the results to be consistent both in terms of the resulting peak signal-to-noise ratios and a visual comparison of the retrieved estimates. Note that our implementation even gives slightly better PSNRs on most images, since it uses a full local search for each reference coordinate instead of reducing the size of the local search window based on previous block-matching results, as it was suggested in [24]. However, due to both this fact and the lack of code optimization, our implementation is currently also slower by a factor of between 5 and 10, depending on the input.
At http://nmevenkamp.github.io/ELMA/, we provide a current version of our proposed method as C++ source code, as well as Windows and MacOSX applications with graphical user interface.
Adaptive non-local periodic block matching
A major goal of this work is to show that local block matching, with a search window small enough to warrant practical efficiency, is not able to properly benefit from globally recurring self-similar features. Instead, in such cases, real non-local block-matching strategies are able to exploit the capabilities of the BM3D filter to a significantly higher extent.
PSNRs of local vs. adaptive periodic block matching
Input | Peak | PSNR [dB] | np-l8 | np-π N8 | np-l16 | np-π N16 | ||
---|---|---|---|---|---|---|---|---|
Silicon (disloc.) | 6 | 2.70 | 17.96 | 18.47 | 15.07 | 17.16 | ||
Silicon | 6 | 3.62 | 20.60 | 21.65 | 20.68 | 23.39 | ||
Gallium nitride | 7 | 6.11 | 21.71 | 23.26 | 20.53 | 24.76 | ||
Silicon | 9 | 6.65 | 22.25 | 23.92 | 22.60 | 26.15 | ||
Silicon | 11 | 9.63 | 23.59 | 24.02 | 25.23 | 26.00 | ||
Silicon (disloc.) | 11 | 9.72 | 23.78 | 24.39 | 25.30 | 26.64 | ||
Gallium nitride | 12 | 10.15 | 24.30 | 26.13 | 24.72 | 27.90 | ||
Silicon | 18 | 12.58 | 25.76 | 26.74 | 26.65 | 27.99 | ||
Silicon (disloc.) | 15 | 12.73 | 25.68 | 26.34 | 26.64 | 27.65 | ||
Gallium nitride | 17 | 13.25 | 26.18 | 27.86 | 26.51 | 29.04 | ||
Silicon | 21 | 13.64 | 26.51 | 27.80 | 26.62 | 28.86 | ||
Silicon | 21 | 15.12 | 27.05 | 27.74 | 27.91 | 28.76 | ||
Silicon | 34 | 16.68 | 28.23 | 29.53 | 28.25 | 30.25 | ||
Silicon (disloc.) | 49 | 19.81 | 29.49 | 29.94 | 29.90 | 30.20 | ||
Silicon (disloc.) | 89 | 22.66 | 31.16 | 31.48 | 31.53 | 31.77 | ||
Silicon (disloc.) | 92 | 22.73 | 31.14 | 31.39 | 31.50 | 31.67 | ||
Silicon | 88 | 22.75 | 31.25 | 31.56 | 31.60 | 31.80 | ||
Silicon | 88 | 22.88 | 31.40 | 31.74 | 31.78 | 31.99 | ||
Gallium nitride | 95 | 23.23 | 32.28 | 33.58 | 32.30 | 33.89 |
Uniform distribution of block estimates
PSNRs of periodic vs. uniform periodic block matching
Input | Peak | PSNR [dB] | np-π N8 | np-π U8 | np-π N16 | np-π U16 | ||
---|---|---|---|---|---|---|---|---|
Silicon | 6 | 3.62 | 21.65 | 22.14 | 23.39 | 23.80 | ||
Silicon | 9 | 6.65 | 23.92 | 24.21 | 26.15 | 26.21 | ||
Silicon | 11 | 9.63 | 24.02 | 24.19 | 26.00 | 25.62 | ||
Silicon | 18 | 12.58 | 26.74 | 26.79 | 27.99 | 28.02 | ||
Silicon | 21 | 15.12 | 27.74 | 27.81 | 28.76 | 28.75 | ||
Silicon | 21 | 13.64 | 27.80 | 27.94 | 28.86 | 28.89 | ||
Silicon | 34 | 16.68 | 29.53 | 29.56 | 30.25 | 30.28 | ||
Silicon | 88 | 22.75 | 31.56 | 31.68 | 31.80 | 31.94 | ||
Silicon | 88 | 22.88 | 31.74 | 31.84 | 31.99 | 32.15 | ||
Gallium nitride | 7 | 6.11 | 23.26 | 23.78 | 24.76 | 24.82 | ||
Gallium nitride | 12 | 10.15 | 26.13 | 26.49 | 27.90 | 27.99 | ||
Gallium nitride | 17 | 13.25 | 27.86 | 28.09 | 29.04 | 28.99 | ||
Gallium nitride | 95 | 23.23 | 33.58 | 33.48 | 33.89 | 33.58 | ||
Silicon (disloc.) | 6 | 2.70 | 18.47 | 19.53 | 17.16 | 19.50 | ||
Silicon (disloc.) | 11 | 9.72 | 24.39 | 24.88 | 26.64 | 26.55 | ||
Silicon (disloc.) | 15 | 12.73 | 26.34 | 26.50 | 27.65 | 27.44 | ||
Silicon (disloc.) | 49 | 19.81 | 29.94 | 29.87 | 30.20 | 30.14 | ||
Silicon (disloc.) | 89 | 22.66 | 31.48 | 31.48 | 31.77 | 31.75 | ||
Silicon (disloc.) | 92 | 22.73 | 31.39 | 31.44 | 31.67 | 31.67 |
Direct block matching for Poisson noise statistics
PSNRs of Anscombe andL^{2} distance based block similarity vs. Poisson maximum-likelihood ratios
Input | Peak | PSNR [dB] | np-π N8 | pml-π N8 | np-π N16 | pml-π N16 | ||
---|---|---|---|---|---|---|---|---|
Silicon (disloc.) | 6 | 2.70 | 18.46 | 18.59 | 17.29 | 17.48 | ||
Silicon | 6 | 3.62 | 21.63 | 21.81 | 23.41 | 23.59 | ||
Gallium nitride | 7 | 6.11 | 23.24 | 23.25 | 24.80 | 24.84 | ||
Silicon | 9 | 6.65 | 23.92 | 24.07 | 26.15 | 26.22 | ||
Silicon | 11 | 9.63 | 24.02 | 24.06 | 26.00 | 25.92 | ||
Silicon (disloc.) | 11 | 9.72 | 24.39 | 24.50 | 26.64 | 26.55 | ||
Gallium nitride | 12 | 10.15 | 26.13 | 26.19 | 27.90 | 27.91 |
Although the improvement so far is small, we have just modified a small part of the BM3D algorithm to work directly on the original Poisson statistics, namely the block-matching part within the initial hard thresholding iteration. The results we presented in [20] show a more significant advantage of the Poisson maximum-likelihood ratio-based similarity measure over the Anscombe transformed L^{2}-distances. However, this was for the NLM algorithm, where in the case of the Poisson maximum-likelihood ratio-based similarity measure, no Anscombe transform of the input data was required at all. Therefore, we believe that future research towards adopting the whole BM3D algorithm directly to Poisson noise statistics may lead to more substantial improvements of the reconstruction in the extreme low-count regime.
Atomic column detectability, position precision, and reconstruction fidelity
As mentioned in the beginning, the positions of the atoms are an important quantity that material scientists would like to extract from atomic-resolution electron micrographs. In the following, we analyze how well the positions can be estimated on the noisy images referred to in the previous sections and by how much our proposed denoising algorithm improves this estimation.
In order to extract the atom positions, we adopt a two-step procedure: first, the individual atoms within the image are detected through segmentation and the geometrical centers of the resulting atomic regions are used as an initial guess for the atom centers; second, a 2D Gaussian function (or a sum of two 2D Gaussian functions in the case of the silicon lattices) is fit on a small area around each atomic region via non-linear regression in order to determine the final estimate of the position of each atom. For further details regarding this procedure, we refer to [2,33].
Detection fraction (26) of the noisy (denoised) images
Input | Peak | Dose [C/cm^{2}] | ρ_{+} | np-l16 | np-π N16 | np-π U16 |
---|---|---|---|---|---|---|
Silicon (disloc.) | 6 | 0.5476 | 0.0238 | 0 | 0.4841 | 0.9246 |
Silicon | 6 | 0.7853 | 0.0192 | 0.9231 | 0.9423 | 0.9808 |
Gallium nitride | 7 | 1.5400 | 0.1333 | 0.1444 | 1 | 1 |
Silicon | 9 | 1.5706 | 0.2203 | 1 | 1 | 1 |
Silicon (disloc.) | 11 | 2.8064 | 0 | 0.9927 | 0.9927 | 0.9927 |
Gallium nitride | 12 | 3.8499 | 0.0105 | 0.9684 | 0.9895 | 1 |
Silicon (disloc.) | 15 | 5.6812 | 0.9818 | 1 | 1 | 1 |
Silicon | 21 | 7.8530 | 1 | 1 | 1 | 1 |
Silicon | 34 | 15.7061 | 1 | 1 | 1 | 1 |
Silicon (disloc.) | 49 | 28.4745 | 1 | 1 | 1 | 1 |
Misdetection fraction (27) of the noisy (denoised) images
Input | Peak | Dose [C/cm^{2}] | ρ_{−} | np-l16 | np-π N16 | np-π U16 |
---|---|---|---|---|---|---|
Silicon (disloc.) | 6 | 0.5476 | 0.4000 | 1 | 0.1223 | 0.0412 |
Silicon | 6 | 0.7853 | 0.8000 | 0.0204 | 0 | 0 |
Gallium nitride | 7 | 1.5400 | 0.2000 | 0 | 0 | 0 |
Silicon | 9 | 1.5706 | 0.2353 | 0 | 0 | 0 |
Silicon (disloc.) | 11 | 2.8064 | n/a | 0 | 0 | 0 |
Gallium nitride | 12 | 3.8499 | 0 | 0 | 0 | 0 |
Silicon (disloc.) | 15 | 5.6812 | 0.0218 | 0 | 0 | 0 |
Silicon | 21 | 7.8530 | 0 | 0.0189 | 0 | 0 |
Silicon | 34 | 15.7061 | 0 | 0 | 0 | 0 |
Silicon (disloc.) | 49 | 28.4745 | 0 | 0 | 0 | 0 |
These specify the fraction of atoms detected from the noisy (denoised) image that could not be matched to any of the atoms detected from the ground truth. As seen in Table 5, these fractions are zero, except for the noisiest input images (6 to 15 peak electrons/pixel) and the local block-matching BM3D estimate of the silicon images with doses 0.79 and 7.85 C/cm ^{2}, as well as the lowest dose silicon dislocation image (0.55 C/cm ^{2}). Note that for the periodic block-matching BM3D the misdetection fractions are zero in all cases, except for the lowest dose silicon dislocation image. While the local block-matching BM3D has a misdetection fraction of 100% in this case, the periodic and uniform periodic block-matching BM3D methods achieve misdetection fractions of 12% and 4%, respectively.
Precision (28), in picometers, of the noisy and denoised images
Input | Peak | Dose [C/cm^{2}] | Precision [pm] | np-l16 | np-π N16 | np-π U16 |
---|---|---|---|---|---|---|
Silicon | 6 | 0.7853 | n/a | 28.06 | 12.18 | 14.34 |
Gallium nitride | 7 | 1.5400 | n/a | n/a | 11.03 | 10.63 |
Silicon | 9 | 1.5706 | n/a | 21.38 | 9.63 | 9.22 |
Gallium nitride | 12 | 3.8499 | n/a | 9.98 | 7.65 | 7.26 |
Silicon | 21 | 7.8530 | 13.79 | 10.23 | 4.65 | 4.85 |
Silicon | 34 | 15.7061 | 9.30 | 8.70 | 4.41 | 4.61 |
Fidelity (29), in picometers, between ground truths and noisy (denoised) images
Input | Peak | Dose [C/cm^{2}] | Fidelity [pm] | np-l16 | np-π N16 | np-π U16 |
---|---|---|---|---|---|---|
Silicon (disloc.) | 6 | 0.5476 | 17.81 | n/a | 2.71 | 1.66 |
Silicon | 6 | 0.7853 | 8.59 | 2.77 | 1.32 | 1.29 |
Gallium nitride | 7 | 1.5400 | 17.09 | 3.43 | 1.16 | 1.16 |
Silicon | 9 | 1.5706 | 7.75 | 1.57 | 0.74 | 0.73 |
Silicon (disloc.) | 11 | 2.8064 | n/a | 0.65 | 0.50 | 0.54 |
Gallium nitride | 12 | 3.8499 | 5.54 | 0.70 | 0.59 | 0.57 |
Silicon (disloc.) | 15 | 5.6812 | 1.26 | 0.49 | 0.38 | 0.43 |
Silicon | 21 | 7.8530 | 0.98 | 0.73 | 0.35 | 0.37 |
Silicon | 34 | 15.7061 | 0.56 | 0.53 | 0.28 | 0.28 |
Silicon (disloc.) | 49 | 28.4745 | 0.42 | 0.30 | 0.27 | 0.29 |
A comparison with linear filters
In practice, simple linear filters, such as the median or low-pass Wiener filter, are still commonly used for denoising in electron microscopy. In the following, we show that such filters provide a substantially worse denoising performance than the proposed modified BM3D method. In particular, we demonstrate that the application of such linear filters does not enable a proper analysis of extremely noisy images (less than 10 peak electron counts per pixel). We used the MATLAB functions medfilt2 for median filtering and wiener2 for low-pass Wiener filtering to produce the results presented here. In both cases, default parameters were used, which results in a windows size of 3×3 pixels and, in case of the Wiener filter, the noise power being automatically estimated.
Performance of median and Wiener filters
Input | Peak | Dose [C/cm^{2}] | ρ_{+} | ρ_{−} | Precision [pm] | Fidelity [pm] | |
---|---|---|---|---|---|---|---|
Median | Silicon | 6 | 0.7853 | 0.96 | 0.81 | 38.73 | 2.86 |
Gallium nitride | 7 | 1.5400 | 1 | 0 | 19.01 | 1.36 | |
Silicon | 9 | 1.5706 | 0.83 | 0.06 | 29.81 | 2.46 | |
Gallium nitride | 12 | 3.8499 | 1 | 0 | 12.07 | 0.74 | |
Wiener | Silicon | 6 | 0.7853 | 0.04 | 0.5 | n/a | 9.44 |
Gallium nitride | 7 | 1.5400 | 0.19 | 0.06 | n/a | 14.69 | |
Silicon | 9 | 1.5706 | 0.02 | 0.67 | n/a | 12.81 | |
Gallium nitride | 12 | 3.8499 | 0 | n/a | n/a | n/a |
A remark on further quantitative measures
We are aware that other quantities like the intensity and shape of atomic columns are also important for materials science. In future work, we plan to conduct a survey in this direction, especially treating the question whether the non-local averaging procedure will be able to retain different intensities in atomic columns of the same type, thus enabling the determination of the number of atoms in each column within the denoised image.
Conclusions
We proposed key modifications of the block-matching and 3D-filtering algorithm, which were aimed at enhancing the filter when applied to atomic-resolution electron micrographs of periodic crystals. We have shown that, through the proposed modifications, the denoising performance is significantly improved compared to the original BM3D on all tested images. It also substantially outperforms common linear filters such as median-filtering and low-pass Wiener filtering. The major advances are the adoption of a Fourier-based periodic similarity search [20] within the non-local means setting to the BM3D algorithm, as well as the treatment of an issue regarding spatial block concentration, which only occurs in the new BM3D setting. Furthermore, we showed that the proposed filter with its uniform adaptive periodic block matching, specifically tailored to perfect crystal structures, is able to significantly enhance both visually and quantitatively the image quality of low-dose electron micrographs. Quantitative measures of interest to the material science community, namely atomic column detectability and position precision, are significantly improved by application of the new denoising algorithm, without the introduction of artifacts such as false-positive identification of atomic columns or shifts in the atomic column image positions beyond the sub-picometer level.
The proposed algorithm for steering the spatial distribution of block estimates towards global uniformity achieves a significant improvement over the periodic block-matching BM3D for certain images. However, we also observed cases where the results are slightly worse compared to the periodic block matching without this addition. Note that our uniform adaptive periodic block-matching BM3D is still significantly better than the original BM3D in all cases. Nevertheless, we plan to further investigate this phenomenon. The goal is to find a strategy that will at least sustain the quality of the estimate compared to plain periodic block matching while improving it in the majority of the cases.
According to the results we presented for the case of a silicon dislocation, the adaptiveness of the periodic block matching copes fairly well with localized irregularities in the crystal structure. Nevertheless, the proposed periodic block matching is generally limited to the assumption of a perfectly periodic crystal, which is usually of less interest to material scientists than crystals with (possibly multiple) defects or changes in the lattice orientation. Thus, we plan to improve and properly extend the block-matching strategy to these more complex geometries.
While the presented methods can be easily adopted to a series of images and thus might directly enable the registration of a series of low-dose electron micrographs, we expect a direct coupling of denoising and registration to be superior. Developing such a combined algorithm and analyzing its performance on series of low dose frames will be a main goal of our future work.
Notes
Acknowledgements
Electron microscopy simulations and other work by ABY and PMV were supported by the US Department of Energy, Office of Basic Energy Sciences, Division of Materials Sciences and Engineering under Award #DE-FG02-08ER46547. PB and WD acknowledge funding from the National Science Foundation under Grant No. DMS-1222390. The authors at RWTH Aachen were funded in part by the Excellence Initiative of the German Federal and State Governments.
References
- 1.Batson, PE, Dellby, N, Krivanek OL: Sub-ångstrom resolution using aberration corrected electron optics. Nature. 418(6898), 617–620 (2002).CrossRefGoogle Scholar
- 2.Berkels, B, Binev, P, Blom, DA, Dahmen, W, Sharpley, R, Vogt, T: Optimized imaging using non-rigid registration. Ultramicroscopy. 138, 46–56 (2014).CrossRefGoogle Scholar
- 3.Ortalan, V, Uzan, A, Gates, BC, Browning, ND: Direct imaging of single metal atoms and clusters in the pores of dealuminated HY zeolite. Nat. Nanotechnol. 5, 506–510 (2010).CrossRefGoogle Scholar
- 4.Houben, L, Thust, A, Urban, K: Atomic-precision determination of the reconstruction of a 90 degree tilt boundary in YBa _{2} Cu _{3} O _{7−δ} by aberration corrected HRTEM. Ultramicroscopy. 106(3), 200–214 (2006).CrossRefGoogle Scholar
- 5.Jia, C-L, Urban, K, Alexe, M, Hesse, D, Vrejoiu, I: Direct observation of continuous electric dipole rotation in flux-closure domains in ferroelectric Pb(Zr,Ti)O _{3}. Science. 331(6023), 1420–1423 (2011).CrossRefGoogle Scholar
- 6.Schmid, H, Okunishi, E, Oikawa, T, Mader, W: Structural and elemental analysis of iron and indium doped zinc oxide by spectroscopic imaging in cs-corrected. STEM Micron. 43(1), 49–56 (2012).CrossRefGoogle Scholar
- 7.Kimoto, K, Asaka, T, Yu, X, Nagai, T, Matsui, Y, Ishizuka, K: Local crystal structure analysis with several picometer precision using scanning transmission electron microscopy. Ultramicroscopy. 110(7), 778–782 (2010).CrossRefGoogle Scholar
- 8.Yankovich, AB, Berkels, B, Dahmen, W, Binev, P, Sanchez, SI, Bradley, SA, Li, A, Szlufarska, I, Voyles, PM: Picometre-precision analysis of scanning transmission electron microscopy images of platinum nanocatalysts. Nat. Commun (2014). doi:10.1038/ncomms5155.Google Scholar
- 9.Nellist, PD, Pennycook, SJ: Accurate structure determination from image reconstruction in adf stem. J. Microsc. 190(1-2), 159–170 (1998).CrossRefGoogle Scholar
- 10.Rosenauer, A, Mehrtensx, T, Müller, K, Gries, K, Schowalter, M, Venkata Satyam, P, Bley, S, Tessarek, C, Hommel, D, Sebald, K, Seyfried, M, Gutowski, J, Avramescu, A, Engl, K, Lutgen, S: Composition mapping in ingan by scanning transmission electron microscopy. Ultramicroscopy. 111(8), 1316–1327 (2011).CrossRefGoogle Scholar
- 11.Kim, Y-M, He, J, Biegalski, MD, Ambaye, H, Lauter, V, Christen, HM, Pantelides, ST, Pennycook, SJ, Kalinin, SV, Borisevich, AY: Probing oxygen vacancy concentration and homogeneity in solid-oxide fuel-cell cathode materials on the subunit-cell level. Nat. Mater. 11(10), 888–894 (2012).CrossRefGoogle Scholar
- 12.Morgan, DG, Ramasse, QM, Browning, ND: Application of two-dimensional crystallography and image processing to atomic resolution z-contrast images. J. Electron Microsc. 58(3), 223–244 (2009).CrossRefGoogle Scholar
- 13.Unwin, PNT, Henderson, R: Molecular structure determination by electron microscopy of unstained crystalline specimens. J. Mol. Biol. 94(3), 425–440 (1975).CrossRefGoogle Scholar
- 14.Klug, A: Image analysis and reconstruction in the electron microscopy of biological macromolecules. Chemica Scripta. 14, 1978–1979.Google Scholar
- 15.Hovmöller, S, Sjögren, A, Farrants, G, Sundberg, M, Marinder, B-O: Accurate atomic positions from electron microscopy. Nature. 311, 238–241 (1984).CrossRefGoogle Scholar
- 16.Hovmöller, S: Crisp: crystallographic image processing on a personal computer. Ultramicroscopy. 41(1), 121–135 (1992).CrossRefGoogle Scholar
- 17.Buades, A, Coll, B, Morel, J-M: A review of image denoising algorithms, with a new one. Multiscale Model. Simul. 4(2), 490–530 (2005).CrossRefGoogle Scholar
- 18.Binev, P, Blanco-Silva, F, Blom, D, Dahmen, W, Lamby, P, Sharpley, R, Vogt, T: Modeling Nanoscale Imaging in Electron Microscopy. Nanostructure Science and Technology. In: Vogt, T, Dahmen, W, Binev, P (eds.), pp. 127–145. Springer, New York City (2012).Google Scholar
- 19.Wei, D-Y, Yin, C-C: An optimized locally adaptive non-local means denoising filter for cryo-electron microscopy data. J. Struct. Biol. 172(3), 211–218 (2010).CrossRefGoogle Scholar
- 20.Mevenkamp, N, Yankovich, AB, Voyles, PM, Berkels, B: Non-local means for scanning transmission electron microscopy images and poisson noise based on adaptive periodic similarity search and patch regularization. In: Bender, J, Kuijper, A, von Landesberger, T, Theisel, H, Urban, P (eds.)VMV 2014: Vision, Modeling & Visualization, pp. 63–70. Eurographics Association, Darmstadt, Germany (2014).Google Scholar
- 21.Bergmann, R: The fast Fourier transform and fast wavelet transform for patterns on the torus. Appl. Comput. Harmonic Anal. 35(1), 39–51 (2013).CrossRefGoogle Scholar
- 22.Coupé, P, Yger, P, Prima, S, Hellier, P, Kervrann, C, Barillot, C: An Optimized Blockwise Nonlocal Means Denoising Filter for 3-D Magnetic Resonance Images. IEEE Trans. Med. Imaging. 27(4), 425–441 (2008).CrossRefGoogle Scholar
- 23.Salmon, J, Harmany, Z, Deledalle, CA, Willett, R: Poisson noise reduction with non-local PCA. J. Math. Imaging Vis. 48(2), 279–294 (2013).CrossRefGoogle Scholar
- 24.Dabov, K, Foi, A, Katkovnik, V, Egiazarian, K: Image Denoising by sparse 3-D transform-domain collaborative filtering. Image Process. IEEE Trans. 16(8), 2080–2095 (2007).CrossRefGoogle Scholar
- 25.Anscombe, FJ: The transformation of poisson, binomial and negative-binomial data. Biometrika Trust. 35, 246–254 (1948).CrossRefGoogle Scholar
- 26.Mäkitalo, M, Foi, A: Optimal inversion of the anscombe transformation in low-count poisson image denoising. Image Process. IEEE Trans. 20(1), 99–109 (2011).CrossRefGoogle Scholar
- 27.Deledalle, C-A, Tupin, F, Denis, L: Poisson NL means: Unsupervised non local means for Poisson noise. In: Paper presented at the 17th Internal Conference on Image Processing, pp. 801–804. IEEE, Hong Kong (26–29 September 2010).Google Scholar
- 28.Dabov, K, Foi, A, Katkovnik, V, Egiazarian, K: Image denoising with block-matching and 3D filtering. Proc. SPIE. 6064, 606414–60641412 (2006).Google Scholar
- 29.Maggioni, M, Sánchez-Monge, E, Foi, A, Danielyan, A, Dabov, K, Katkovnik, V, Egiazarian, K: Image and Video Denoising by Sparse 3D Transform-domain Collaborative Filtering: Block-matching and 3D Filtering (BM3D) Algorithm and Its Extensions. Tampere University of Technology, http://www.cs.tut.fi/~foi/GCF-BM3D/ Accessed date November, 10th 2014.
- 30.Mäkitalo, M, Foi, A: Optimal Inversion of the Anscombe and Generalized Anscombe Variance-stabilizing Transformations. Tampere University of Technology. http://www.cs.tut.fi/~foi/invansc/ Accessed date November, 10th 2014.
- 31.Martin, DC, Chen, J, Yang, J, Drummy, LF, Kübel, C: High resolution electron microscopy of ordered polymers and organic molecular crystals: Recent developments and future possibilities. J. Polymer Sci. Part B: Polymer Phys. 43(14), 1749–1778 (2005).CrossRefGoogle Scholar
- 32.Kirkland, EJ: Advanced Computing in Electron Microscopy. 2nd edn. Springer, New York City (2010).CrossRefGoogle Scholar
- 33.Van Aert, S, Den Dekker, AJ, Van Den Bos, A, Van Dyck, D, Chen, JH: Maximum likelihood estimation of structure parameters from high resolution electron microscopy images. Part II: a practical example. Ultramicroscopy. 104(2), 107–125 (2005).CrossRefGoogle Scholar
Copyright information
This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited.