Poisson noise removal from highresolution STEM images based on periodic block matching
 2.9k Downloads
 10 Citations
Abstract
Scanning transmission electron microscopy (STEM) provides subångstrom, atomic resolution images of crystalline structures. However, in many applications, the ability to extract information such as atom positions, from such electron micrographs, is severely obstructed by low signaltonoise ratios of the acquired images resulting from necessary limitations to the electron dose. We present a denoising strategy tailored to the special features of atomicresolution electron micrographs of crystals limited by Poisson noise based on the blockmatching and 3Dfiltering (BM3D) algorithm by Dabov et al. We also present an economized blockmatching strategy that exploits the periodic structure of the observed crystals. On simulated singleshot STEM images of inorganic materials, with incident electron doses below 4 C/cm ^{2}, our new method achieves precisions of 7 to 15 pm and an increase in peak signaltonoise ratio (PSNR) of 15 to 20 dB compared to noisy images and 2 to 4 dB compared to images denoised with the original BM3D.
Keywords
Image denoising Poisson noise Nonlocal means Block matching 3D transform shrinkage Periodic search STEM PrecisionAbbreviations
 STEM
scanning transmission electron microscopy
 BM3D
blockmatching and 3Dfiltering algorithm
 PSNR
peak signaltonoise ratio
 SNR
signaltonoise ratio
 NLM
nonlocal means
 AGWN
additive Gaussian white noise
 HRTEM
highresolution transmission electron microscopy
 VST
variancestabilizing transform
Background
Modern electron microscopy allows for atomic resolution images of crystalline structures [1]. However, the full scope of resolution can be exploited only for materials with little electron beam sensitivity. Lowering the electron dose decreases the signaltonoise ratio (SNR) of the acquired micrographs accordingly, degrading the quality or even completely prohibiting the extraction of desired information from the noisy micrographs. Examples of inorganic materials with high beam sensitivity, where scanning transmission electron microscopy (STEM) images of poor SNR have to be used, are both oxide [2] and metallic [3] catalysts. One important quantity that may be extracted from atomicresolution electron micrographs is the positions of the atoms. The precision with which these can be determined is crucial for the understanding of certain material properties [4,5]. For singleshot STEM images, the best reported precision is about 15 pm [6]. A common technique to increase the precision is to register a series of frames and extract the atom positions from the average of the registered frames. Kimoto et al. [7] achieved a precision of 5 pm using rigid registration. More recently, a nonrigid registration scheme has been developed by Berkels et al. [2] that has achieved subpicometer precision for STEM series [8]. Both registration schemes require many individual frames of moderate SNR resulting in a veryhigh overall electron dose applied to the material. Thus, to widen the applicability of STEM to more beamsensitive inorganic materials, a central objective is to develop effective denoising methods that increase the singleshot image quality. This would enable the extraction of desired information from individual frames or the use of the individual frames for registration, while using a lower electron dose.
Denoising in materials, electron microscopy is often accomplished by simple spatial filters like a Gaussian blur or median filter or frequency space lowpass or Wiener filters (cf. [911]). These examples are all taken from materials robust enough under the electron beam to survive relatively high electron dose to obtain images with high spatial oversampling (small pixels). More sophisticated approaches that take advantage of the highly redundant nature of some highresolution STEM images have been employed [12], building on tools developed for very beamsensitive biological samples [1316]. While these methods, mostly based on manipulations of the image in Fourier space, are well established and have proven to perform well on highresolution transmission electron microscopy (HRTEM) images of organic materials, they are tightly linked to the periodicity of the data and have thus not been widely adopted in STEM studies of aperiodic defects in inorganic materials.
The most successful generic image denoising method for arbitrary images available today rely on nonlocal detection and averaging of selfsimilar image regions. The first algorithm based on this strategy is the nonlocal means filter (NLM) by Buades et al. [17]. Due to the richness in selfsimilarity of electron micrographs of crystals, NLM is in principle very well suited for denoising such micrographs [18], and it has been employed in biological electron microscopy [19]. However, two main properties of lowdose electron micrographs of crystals are disregarded by the original design and implementation of NLM, namely (1) NLM removes additive Gaussian white noise (AGWN) instead of Poisson noise, which is the dominant form of noise in lowdose STEM images; and (2) NLM uses a local similarity search to reduce the computational cost, whereas the distance between selfsimilar points in images of crystals is at least as large as the interatomic distance. In the case of STEM, there is an additional difficulty caused by image distortions resulting from the serial acquisition of the pixel data. In view of these issues, we proposed an enhanced version of NLM tailored to STEM imaging [20]. The most significant enhancement was the development of an efficient nonlocal similarity search, based on the generation of periodic lattices in Fourier space [21]. However, the basic NLM principle can be further improved by replacing the simple weighted average of intensities of pixels with similar neighborhoods by a more advanced collaborative filtering of the corresponding image parts. Several methods of this type have been proposed over the past few years, e.g., optimized blockwise NLM by Coupé et al. [22] and NLPCA by Salmon et al. [23], which is based on principal component analysis. Among the most successful variants is the blockmatching and 3Dfiltering algorithm (BM3D) developed by Dabov et al. [24], which has become a benchmark for image denoising algorithms in the field of image processing. The extension of BM3D to Poisson noise removal via application of the Anscombe variancestabilizing transformation [25] has already been proposed by Mäkitalo and Foi [26] who have contributed an exact unbiased inverse transformation that increases the accuracy especially in the lowcount regime. In [20], it was found that despite being restricted to a local similarity search, the original BM3D filter with extension to Poisson noise outperformed the proposed periodic search NLM filter. Thus, the starting point here is the BM3D filter.
In this paper, we discuss how modifications introduced in STEMtailored NLM [20] can be incorporated into the stateoftheart denoising algorithm BM3D in order to tap its full potential on electron micrographs of crystals. More precisely, as this method has been designed for Gaussian white noise, a central task is to develop suitable modifications that effectively deal with Poisson noise. Another focus is to exploit the atomic lattice structure that entails a repeated appearance of selfsimilar image components within a single frame. While we focus on the application of STEM imaging of inorganic materials, the key features of Poisson noise and oversampling are shared by HRTEM images as well, so the proposed method should also be applicable to HRTEM images. Also note that the procedures proposed in this paper can be applied subsequently for registering a series of lowdose frames to a single reference frame [2].
The paper is organized as follows. First, we briefly recall the original BM3D algorithm. Then, we present two strategies for Poisson noise removal, namely variance stabilization based on the Anscombe transform and using Poisson maximumlikelihood similarity measures due to Deledalle et al. [27] for the block matching. The main contribution is an adaptive nonlocal periodic block matching. It exploits the repetitive structure within the micrograph while keeping the computational cost affordable. Since the spatial distribution of the resulting block estimates is highly nonuniform, potentially resulting in a poor reconstruction in regions with few available block estimates, we discuss possible remedies of this deficiency. Finally, we evaluate the proposed method on simulated data, where the ground truth, i.e., the true, noisefree mean electron count per pixel, is available, showing the improvement both in terms of visual image quality and in quantitative measures such as peak signaltonoise ratio (PSNR), precision, and fidelity achieved by the proposed method.
Methods
Block matching and 3D filtering
BM3D was developed on the basis of NLM. It was originally designed to remove additive Gaussian white noise from natural images or other images exhibiting a sufficient amount of selfsimilarity. The main idea of the algorithm is to find similar parts within an image and remove noise from those parts by sparsifying their common representation in a 3D transform domain. After processing all image parts in this manner, one receives an overcomplete representation of an estimate of the noisefree input image. Averaging all partial estimates in overlapping regions provides an estimate of the full underlying ground truth image. The denoising process in the 3D transform domain is performed in two stages, namely an initial hard thresholding succeeded by a final Wiener filtering. In the following, we briefly recall the definition of the original BM3D filter as published in [28], which serves as a basis for the methods developed in this work. We use a slightly simplified version of the algorithm described in [28] but with the exception of a change of the block size from 8×8 to 16×16, all relevant parameters, as well as the unitary transforms in the 3D domain, are chosen in exact agreement with the original implementation of the BM3D filter [29] when using the default profile, i.e., normal profile.
The original BM3D algorithm for Gaussian noise removal
where f is the (noisefree) ground truth and \( \eta \left(\cdotp \right)\sim \mathcal{N}\left(0,{\sigma}^2\right) \) is AGWN, i.e., a normally distributed random variable with zero mean and standard deviation σ.
for the tensor \( {\mathcal{Z}}_x \). A typical transformation would be based, e.g., on a wavelet basis. To this estimator, one assigns a weight \( {w}_x={N}_{\theta}^{1} \), where N _{ θ } is the number of retained nonzero transform coefficients after hard thresholding.
where χ _{ Z } denotes the characteristic function of the block \( Z\subset X \). Note that each block Z _{ y } may be matched to different references \( {Z}_x,{Z}_{x^{\prime }} \), resulting in different estimates \( {\hat{Y}}_{x,k},{\hat{Y}}_{x^{\prime },{k}^{\prime }} \) for the same block, i.e., \( {\left({\hat{S}}_x\right)}_k={\left({\hat{S}}_{x^{\prime }}\right)}_{k^{\prime }}=y \) in this case.
The final estimate \( \widehat{f} \) of the desired image f is now calculated in a second iteration, where the whole procedure described above is repeated with some modifications briefly indicated next, referring to [28] for the details. Denoting by F _{ x }, the n×n block with upperleft corner x extracted now from the basic estimate \( {\widehat{f}}^{\mathrm{basic}} \), the above procedure is applied to the blocks F _{ x } in place of Z _{ x }, where the analogs to the sets S _{ x } in (3) are formed, however, with respect to a possibly different threshold parameter τ ^{′}. Moreover, the analog to (6) may involve a different unitary transformation (e.g., using local cosine transforms) and, perhaps more importantly, hard thresholding is replaced by Wiener filtering. Again, we refer to [28] for the details.
Figure 1 illustrates the BM3D procedure using a STEM image as an example. It demonstrates the three main steps of BM3D, which are performed for each reference block: 1) matching blocks to a reference and stacking the matched blocks into a 3D tensor, 2) denoising this column of matched blocks, and 3) aggregating the resulting denoised blocks at their original positions and averaging overlapping parts of adjacent blocks (green). For simplicity, the illustration uses simple averaging along the column of matched blocks (black dotted arrow) as an example procedure for denoising the 3D tensors. In practice, the approaches described above provide better performance.
Like any blockaveraging or nonlocal means approach, our method has reduced performance near the edges of the image. Since N _{step}<n for every y∈X, there is a block with corner x∈X _{ R } such that y∈Z _{ x }. In other words, for any pixel in the image (even at the boundary), there is a block containing this pixel, resulting in an estimated intensity of the pixel after the corresponding block has been denoised in 3D transform domain. But because there exist fewer overlapping blocks near the image boundaries, there are fewer available estimates of boundary pixels that can be used for further averaging. Therefore, both the basic and final estimates \( {\widehat{f}}^{\mathrm{basic}} \) and \( \widehat{f} \), i.e., the intermediate result and the final denoised image, will show a slightly reduced quality towards the image boundaries. However, the denoising of each block together with its matched blocks in 3D transform domain contributes much more to the image quality than the averaging of spatially overlapping blocks. Thus, the reduced quality at the boundaries of images denoised with BM3D is usually not substantial in practice.
Extension to Poisson noise removal via variance stabilization
Here, given random variables V,W, \( \mathbb{E}\left\{V\BigW\right\} \) denote the expectation of V given W. In their implementation [30], Mäkitalo and Foi provide tabulated values of this inverse for arguments within the range [ A(0),100]. For smaller arguments, the inverse is extended by zero, and for higher arguments, the asymptotic unbiased inverse transform is used, see [25].
The procedure for Poisson noise removal is then given by the following threestep procedure: first, apply the forward Anscombe transform on the input image v, receiving an input image z=A(v) with a noise model similar to AGWN; second, apply the BM3D filter to this image z, receiving an estimate \( \widehat{f} \) of E{A(v)λ}; and lastly, apply the exact unbiased inverse Anscombe transform on \( \widehat{f} \) to obtain an estimate \( \widehat{\lambda}:={A}^{1}\left(\widehat{f}\right) \) of λ.
Direct block matching for Poisson noise statistics
As pointed out by Salmon et al., the Anscombe transform is not accurate in the extreme lowcount regime, i.e., if the average number of counts λ(x) is smaller than three for some x∈X (cf. [23]). STEM images with such a small number of counts are generally deemed useless, so they are rarely acquired or published. However, the ability to extract information from extreme lowdose images would be a substantial advantage for a variety of problems, such as characterization of metallic catalyst particles [3] and polymers and molecular crystals [31].
The performance of two variants of nonlocal means on images affected by Poisson noise are compared in [20]. The first one features the Anscombe transform and the standard L ^{2}patch similarity measure, while the second one uses a Poisson maximumlikelihoodbased patch similarity measure from [27] without a preceding VST on the input data. The results show that when the PSNR of the input image is below 10 dB, the Poisson maximumlikelihood ratiobased patch distances outperform the strategy of using standard L ^{2}distances on the Anscombe transformed data.
we have that 0≤pml(k _{1},k _{2})≤1 for any \( {k}_1,{k}_2\in \mathbb{N} \). From this, it follows that also \( \overline{\mathrm{pml}}\left({Z}_x,{Z}_y\right) \), the geometric mean of the maximumlikelihood ratios between the blocks Z _{ x } and Z _{ y } takes values between 0 (very bad match) and 1 (perfect match). Thus, the corresponding threshold τ ^{ P } should be chosen between 0 (no thresholding) and 1 (only accepting identical blocks). We found that the choice τ ^{ P }=0.55 works well in practice and thus we use this value wherever the Poisson maximumlikelihood ratios are employed in this work. The reduced set \( {\check{{S}_{x}^{P}}} \) and the block stack \( {\mathcal{Z}}_x^P \) are defined analogously to \( {\check{S}}_x \) and \( {\mathcal{Z}}_x \), respectively, just replacing S _{ x } by \( {S}_x^P \).
In other words, when using this strategy, the Anscombe transformed data is used to fill the 3D block stacks with the values of the matched blocks but not to determine the positions of the matched blocks in the 3D stacks.
Here, \( {\hat{Y}}_{x,k}^P(y) \) is defined in analogy to (8). Also note that due to the application of the Anscombe transform to the 3D block stacks in (19), the values of the basic estimate (20) after Poisson maximumlikelihoodratiobased block matching are still in the Anscombe transform domain. Thus, the Wiener filtering iteration remains unchanged.
Adaptive nonlocal periodic block matching
Here, α _{1},α _{2} denotethe angles or directions of the periodic pattern within the input image and Δ x _{1},Δ x _{2} the spacings between the selfsimilar objects along those axes and \( {N}_S^{\pi } \) is the size of the small local search windows (in units of pixels). For an automatic estimation strategy of these parameters from the input image, we refer to [20]. Note that the \( \arg \min \) expression causes an adaptive reset of the search pattern that gives some robustness against errors in both the estimation of the grid parameters and slight variations of the periodic pattern within the image.
Here, the super scripts ht and wie refer to the first and second stage involving hard thresholding and Wiener filtering. In this paper, we have used small \( {N}_S^{\pi}\times {N}_S^{\pi } \) local search windows of size \( {N}_S^{\pi }=5 \). We found that this choice retains reasonable computational efficiency, while being large enough (about 0.1 to 0.25 times as small as the atoms in the images we used) to correct the positions of the nonlocal periodic search steps within each atom. \( {\check{\varPi}}_x^{\mathrm{ht}} \) and \( {\check{\varPi}}_x^{\mathrm{wie}} \) are defined analogously to \( {\check{S}}_x \). Note that for crystals at atomic scale the selfsimilarity within the corresponding image is so rich that with a nonlocal search like this, for nearly all x∈X _{ R }, one finds many more than N _{3D} blocks that are similar to the reference at x, i.e., N _{ x }=N _{3D} holds for nearly all x∈X _{ R }.
In order to reduce the computational cost of the nonlocal periodic search, we made the following technical adjustments to its implementation in comparison with its description in [20] and the expression (21). First, for each reference point x among the two directions α _{1} and α _{2}, the corresponding axis with the largest intersection with the image X is declared to be the primary search axis. Then, after performing one step along the primary search axis, nonlocal periodic search steps along both positive and negative directions of the secondary search axis are carried out only until the image boundary is reached. This process is repeated until the image boundary is also reached along the positive and negative direction of the primary search axis. On the one hand, this implementation requires less computational cost than computing the whole set of points within the image that could be reached by steps along either of the two periodicity axes, while on the other hand, it still gives a sufficiently large subset of all of these points. This efficient periodic blockmatching strategy increases the computational cost by a factor of about 1.5 to 2 (depending on the density of the atoms within the image) compared to our implementation of BM3D with local block matching. Note that due to the overhead produced by unoptimized parts of our implementation this relative comparison may not be exact. Nevertheless, we believe that the factor between the runtimes of local and periodic block matching would be of the same order of magnitude in optimized code as well.
Uniform distribution of block estimates
where \( \mathcal{P}:=\left\{{\left({\varPi}_x^{\prime}\right)}_{x\in {X}_R}:{\varPi}_x^{\prime}\subset {\varPi}_x^{\ast },\#\left({\varPi}_x^{\prime}\right)={N}_x,\kern0.60em \forall x\in {X}_R\right\} \) denotes the feasible set. Unfortunately, this optimization problem is of combinatorial type and due to its global coupling of all reference coordinates would be computationally too costly to solve. Hence, we propose the following simplification of the uniform block matching through an iterative procedure.
Here, the reference coordinates have been numbered \( {X}_R=\left({x}^1,\dots, {x}^{\#\left({X}_R\right)}\right) \) and block matching is performed for one after the other in an iterative fashion. In this paper, we use the number of local \( {N}_S^{\pi}\times {N}_S^{\pi } \) search windows as the limit \( {N}_{3\mathrm{D}}^{\ast } \). Note that as the number of search window is dependent on the reference position x∈X _{ R }, so is the new limit \( {N}_{3\mathrm{D}}^{\ast }={N}_{3\mathrm{D}}^{\ast }(x) \), unlike the limit N _{3D} chosen before, which is independent of x. This choice can be motivated as follows: if the adaptive periodic search succeeds in placing the local search windows roughly according to the pattern of the observed crystal, then for a reference pixel within an atom, each local search window within the corresponding similarity search should overlap a similar part of another atom. Therefore, we expect to find at least one wellmatching block per local search window. Note that we define \( {n}_{\mathrm{aggr}}^U \) in analogy to (23) but with \( {\check{\varPi}}_x \) replaced by \( {\check{\varPi}}_x^U \).
Results and discussion

np ⋆: normal profile and Anscombebased threestep procedure

pml ⋆: Poisson maximumlikelihood ratiobased block matching within hard thresholding iteration

⋆l ⋆: local block matching with N _{ S }=39
 ⋆ π⋆: adaptive periodic block matching with \( {N}_S^{\pi }=5 \)

⋆ π N⋆: selection of N _{3D} bestmatched blocks with thresholds N _{3D}=16 (HT) and N _{3D}=32 (Wiener)

⋆ π U⋆: uniformly distributed selection of N _{3D} blocks from \( {\widehat{N}}_{3\mathrm{D}} \) bestmatched blocks as in Algorithm 1


⋆ ⋆n: block size is n×n
This quantitative measure requires the knowledge of the underlying ground truth, i.e., the exact average counts λ(x) in each pixel x∈X, which is not available in experimental STEM images. Therefore, we have simulated STEM images of various materials using the frozen phonon multislice algorithm [32] to obtain representative ground truth micrographs that are free of Poisson noise (cf. Figure 2A,C,E,G). To make these images representative of experimental images, typical image distortions that are caused by instabilities of the sample and electron beam during experimental STEM image acquisition have been artificially introduced to these simulated images. Known material crystallographic data was used to create the gallium nitride and silicon atomic models. Molecular dynamics was used to calculate the silicon dislocation atomic model [8]. In order to compare the algorithms at different noise levels, we scaled the intensities of the ground truth images to different average electron counts per pixel before applying random Poisson noise. This simulates the usage of different beam currents, resulting in different electron doses. Examples of Poisson noise affected versions of the simulated images can be found in Figure 3B,D,F,H.
In this work, we use our implementation of BM3D with local search as the ‘original’ BM3D benchmark. This comparison is justified, since we verified our implementation against the implementation provided by the original authors [29] and found the results to be consistent both in terms of the resulting peak signaltonoise ratios and a visual comparison of the retrieved estimates. Note that our implementation even gives slightly better PSNRs on most images, since it uses a full local search for each reference coordinate instead of reducing the size of the local search window based on previous blockmatching results, as it was suggested in [24]. However, due to both this fact and the lack of code optimization, our implementation is currently also slower by a factor of between 5 and 10, depending on the input.
At http://nmevenkamp.github.io/ELMA/, we provide a current version of our proposed method as C++ source code, as well as Windows and MacOSX applications with graphical user interface.
Adaptive nonlocal periodic block matching
A major goal of this work is to show that local block matching, with a search window small enough to warrant practical efficiency, is not able to properly benefit from globally recurring selfsimilar features. Instead, in such cases, real nonlocal blockmatching strategies are able to exploit the capabilities of the BM3D filter to a significantly higher extent.
PSNRs of local vs. adaptive periodic block matching
Input  Peak  PSNR [dB]  npl8  np π N8  npl16  np π N16  

Silicon (disloc.)  6  2.70  17.96  18.47  15.07  17.16  
Silicon  6  3.62  20.60  21.65  20.68  23.39  
Gallium nitride  7  6.11  21.71  23.26  20.53  24.76  
Silicon  9  6.65  22.25  23.92  22.60  26.15  
Silicon  11  9.63  23.59  24.02  25.23  26.00  
Silicon (disloc.)  11  9.72  23.78  24.39  25.30  26.64  
Gallium nitride  12  10.15  24.30  26.13  24.72  27.90  
Silicon  18  12.58  25.76  26.74  26.65  27.99  
Silicon (disloc.)  15  12.73  25.68  26.34  26.64  27.65  
Gallium nitride  17  13.25  26.18  27.86  26.51  29.04  
Silicon  21  13.64  26.51  27.80  26.62  28.86  
Silicon  21  15.12  27.05  27.74  27.91  28.76  
Silicon  34  16.68  28.23  29.53  28.25  30.25  
Silicon (disloc.)  49  19.81  29.49  29.94  29.90  30.20  
Silicon (disloc.)  89  22.66  31.16  31.48  31.53  31.77  
Silicon (disloc.)  92  22.73  31.14  31.39  31.50  31.67  
Silicon  88  22.75  31.25  31.56  31.60  31.80  
Silicon  88  22.88  31.40  31.74  31.78  31.99  
Gallium nitride  95  23.23  32.28  33.58  32.30  33.89 
Uniform distribution of block estimates
PSNRs of periodic vs. uniform periodic block matching
Input  Peak  PSNR [dB]  np π N8  np π U8  np π N16  np π U16  

Silicon  6  3.62  21.65  22.14  23.39  23.80  
Silicon  9  6.65  23.92  24.21  26.15  26.21  
Silicon  11  9.63  24.02  24.19  26.00  25.62  
Silicon  18  12.58  26.74  26.79  27.99  28.02  
Silicon  21  15.12  27.74  27.81  28.76  28.75  
Silicon  21  13.64  27.80  27.94  28.86  28.89  
Silicon  34  16.68  29.53  29.56  30.25  30.28  
Silicon  88  22.75  31.56  31.68  31.80  31.94  
Silicon  88  22.88  31.74  31.84  31.99  32.15  
Gallium nitride  7  6.11  23.26  23.78  24.76  24.82  
Gallium nitride  12  10.15  26.13  26.49  27.90  27.99  
Gallium nitride  17  13.25  27.86  28.09  29.04  28.99  
Gallium nitride  95  23.23  33.58  33.48  33.89  33.58  
Silicon (disloc.)  6  2.70  18.47  19.53  17.16  19.50  
Silicon (disloc.)  11  9.72  24.39  24.88  26.64  26.55  
Silicon (disloc.)  15  12.73  26.34  26.50  27.65  27.44  
Silicon (disloc.)  49  19.81  29.94  29.87  30.20  30.14  
Silicon (disloc.)  89  22.66  31.48  31.48  31.77  31.75  
Silicon (disloc.)  92  22.73  31.39  31.44  31.67  31.67 
Direct block matching for Poisson noise statistics
PSNRs of Anscombe and L ^{ 2 } distance based block similarity vs. Poisson maximumlikelihood ratios
Input  Peak  PSNR [dB]  np π N8  pml π N8  np π N16  pml π N16  

Silicon (disloc.)  6  2.70  18.46  18.59  17.29  17.48  
Silicon  6  3.62  21.63  21.81  23.41  23.59  
Gallium nitride  7  6.11  23.24  23.25  24.80  24.84  
Silicon  9  6.65  23.92  24.07  26.15  26.22  
Silicon  11  9.63  24.02  24.06  26.00  25.92  
Silicon (disloc.)  11  9.72  24.39  24.50  26.64  26.55  
Gallium nitride  12  10.15  26.13  26.19  27.90  27.91 
Although the improvement so far is small, we have just modified a small part of the BM3D algorithm to work directly on the original Poisson statistics, namely the blockmatching part within the initial hard thresholding iteration. The results we presented in [20] show a more significant advantage of the Poisson maximumlikelihood ratiobased similarity measure over the Anscombe transformed L ^{2}distances. However, this was for the NLM algorithm, where in the case of the Poisson maximumlikelihood ratiobased similarity measure, no Anscombe transform of the input data was required at all. Therefore, we believe that future research towards adopting the whole BM3D algorithm directly to Poisson noise statistics may lead to more substantial improvements of the reconstruction in the extreme lowcount regime.
Atomic column detectability, position precision, and reconstruction fidelity
As mentioned in the beginning, the positions of the atoms are an important quantity that material scientists would like to extract from atomicresolution electron micrographs. In the following, we analyze how well the positions can be estimated on the noisy images referred to in the previous sections and by how much our proposed denoising algorithm improves this estimation.
In order to extract the atom positions, we adopt a twostep procedure: first, the individual atoms within the image are detected through segmentation and the geometrical centers of the resulting atomic regions are used as an initial guess for the atom centers; second, a 2D Gaussian function (or a sum of two 2D Gaussian functions in the case of the silicon lattices) is fit on a small area around each atomic region via nonlinear regression in order to determine the final estimate of the position of each atom. For further details regarding this procedure, we refer to [2,33].
Detection fraction ( 26 ) of the noisy (denoised) images
Input  Peak  Dose [C/cm ^{ 2 } ]  ρ _{ + }  npl16  np π N16  np π U16 

Silicon (disloc.)  6  0.5476  0.0238  0  0.4841  0.9246 
Silicon  6  0.7853  0.0192  0.9231  0.9423  0.9808 
Gallium nitride  7  1.5400  0.1333  0.1444  1  1 
Silicon  9  1.5706  0.2203  1  1  1 
Silicon (disloc.)  11  2.8064  0  0.9927  0.9927  0.9927 
Gallium nitride  12  3.8499  0.0105  0.9684  0.9895  1 
Silicon (disloc.)  15  5.6812  0.9818  1  1  1 
Silicon  21  7.8530  1  1  1  1 
Silicon  34  15.7061  1  1  1  1 
Silicon (disloc.)  49  28.4745  1  1  1  1 
Misdetection fraction ( 27 ) of the noisy (denoised) images
Input  Peak  Dose [C/cm ^{ 2 } ]  ρ _{ − }  npl16  np π N16  np π U16 

Silicon (disloc.)  6  0.5476  0.4000  1  0.1223  0.0412 
Silicon  6  0.7853  0.8000  0.0204  0  0 
Gallium nitride  7  1.5400  0.2000  0  0  0 
Silicon  9  1.5706  0.2353  0  0  0 
Silicon (disloc.)  11  2.8064  n/a  0  0  0 
Gallium nitride  12  3.8499  0  0  0  0 
Silicon (disloc.)  15  5.6812  0.0218  0  0  0 
Silicon  21  7.8530  0  0.0189  0  0 
Silicon  34  15.7061  0  0  0  0 
Silicon (disloc.)  49  28.4745  0  0  0  0 
These specify the fraction of atoms detected from the noisy (denoised) image that could not be matched to any of the atoms detected from the ground truth. As seen in Table 5, these fractions are zero, except for the noisiest input images (6 to 15 peak electrons/pixel) and the local blockmatching BM3D estimate of the silicon images with doses 0.79 and 7.85 C/cm ^{2}, as well as the lowest dose silicon dislocation image (0.55 C/cm ^{2}). Note that for the periodic blockmatching BM3D the misdetection fractions are zero in all cases, except for the lowest dose silicon dislocation image. While the local blockmatching BM3D has a misdetection fraction of 100% in this case, the periodic and uniform periodic blockmatching BM3D methods achieve misdetection fractions of 12% and 4%, respectively.
Precision ( 28 ), in picometers, of the noisy and denoised images
Input  Peak  Dose [C/cm ^{ 2 } ]  Precision [pm]  npl16  np π N16  np π U16 

Silicon  6  0.7853  n/a  28.06  12.18  14.34 
Gallium nitride  7  1.5400  n/a  n/a  11.03  10.63 
Silicon  9  1.5706  n/a  21.38  9.63  9.22 
Gallium nitride  12  3.8499  n/a  9.98  7.65  7.26 
Silicon  21  7.8530  13.79  10.23  4.65  4.85 
Silicon  34  15.7061  9.30  8.70  4.41  4.61 
Fidelity ( 29 ), in picometers, between ground truths and noisy (denoised) images
Input  Peak  Dose [C/cm ^{ 2 } ]  Fidelity [pm]  npl16  np π N16  np π U16 

Silicon (disloc.)  6  0.5476  17.81  n/a  2.71  1.66 
Silicon  6  0.7853  8.59  2.77  1.32  1.29 
Gallium nitride  7  1.5400  17.09  3.43  1.16  1.16 
Silicon  9  1.5706  7.75  1.57  0.74  0.73 
Silicon (disloc.)  11  2.8064  n/a  0.65  0.50  0.54 
Gallium nitride  12  3.8499  5.54  0.70  0.59  0.57 
Silicon (disloc.)  15  5.6812  1.26  0.49  0.38  0.43 
Silicon  21  7.8530  0.98  0.73  0.35  0.37 
Silicon  34  15.7061  0.56  0.53  0.28  0.28 
Silicon (disloc.)  49  28.4745  0.42  0.30  0.27  0.29 
A comparison with linear filters
In practice, simple linear filters, such as the median or lowpass Wiener filter, are still commonly used for denoising in electron microscopy. In the following, we show that such filters provide a substantially worse denoising performance than the proposed modified BM3D method. In particular, we demonstrate that the application of such linear filters does not enable a proper analysis of extremely noisy images (less than 10 peak electron counts per pixel). We used the MATLAB functions medfilt2 for median filtering and wiener2 for lowpass Wiener filtering to produce the results presented here. In both cases, default parameters were used, which results in a windows size of 3×3 pixels and, in case of the Wiener filter, the noise power being automatically estimated.
Performance of median and Wiener filters
Input  Peak  Dose [C/cm ^{ 2 } ]  ρ _{ + }  ρ _{ − }  Precision [pm]  Fidelity [pm]  

Median  Silicon  6  0.7853  0.96  0.81  38.73  2.86 
Gallium nitride  7  1.5400  1  0  19.01  1.36  
Silicon  9  1.5706  0.83  0.06  29.81  2.46  
Gallium nitride  12  3.8499  1  0  12.07  0.74  
Wiener  Silicon  6  0.7853  0.04  0.5  n/a  9.44 
Gallium nitride  7  1.5400  0.19  0.06  n/a  14.69  
Silicon  9  1.5706  0.02  0.67  n/a  12.81  
Gallium nitride  12  3.8499  0  n/a  n/a  n/a 
A remark on further quantitative measures
We are aware that other quantities like the intensity and shape of atomic columns are also important for materials science. In future work, we plan to conduct a survey in this direction, especially treating the question whether the nonlocal averaging procedure will be able to retain different intensities in atomic columns of the same type, thus enabling the determination of the number of atoms in each column within the denoised image.
Conclusions
We proposed key modifications of the blockmatching and 3Dfiltering algorithm, which were aimed at enhancing the filter when applied to atomicresolution electron micrographs of periodic crystals. We have shown that, through the proposed modifications, the denoising performance is significantly improved compared to the original BM3D on all tested images. It also substantially outperforms common linear filters such as medianfiltering and lowpass Wiener filtering. The major advances are the adoption of a Fourierbased periodic similarity search [20] within the nonlocal means setting to the BM3D algorithm, as well as the treatment of an issue regarding spatial block concentration, which only occurs in the new BM3D setting. Furthermore, we showed that the proposed filter with its uniform adaptive periodic block matching, specifically tailored to perfect crystal structures, is able to significantly enhance both visually and quantitatively the image quality of lowdose electron micrographs. Quantitative measures of interest to the material science community, namely atomic column detectability and position precision, are significantly improved by application of the new denoising algorithm, without the introduction of artifacts such as falsepositive identification of atomic columns or shifts in the atomic column image positions beyond the subpicometer level.
The proposed algorithm for steering the spatial distribution of block estimates towards global uniformity achieves a significant improvement over the periodic blockmatching BM3D for certain images. However, we also observed cases where the results are slightly worse compared to the periodic block matching without this addition. Note that our uniform adaptive periodic blockmatching BM3D is still significantly better than the original BM3D in all cases. Nevertheless, we plan to further investigate this phenomenon. The goal is to find a strategy that will at least sustain the quality of the estimate compared to plain periodic block matching while improving it in the majority of the cases.
According to the results we presented for the case of a silicon dislocation, the adaptiveness of the periodic block matching copes fairly well with localized irregularities in the crystal structure. Nevertheless, the proposed periodic block matching is generally limited to the assumption of a perfectly periodic crystal, which is usually of less interest to material scientists than crystals with (possibly multiple) defects or changes in the lattice orientation. Thus, we plan to improve and properly extend the blockmatching strategy to these more complex geometries.
While the presented methods can be easily adopted to a series of images and thus might directly enable the registration of a series of lowdose electron micrographs, we expect a direct coupling of denoising and registration to be superior. Developing such a combined algorithm and analyzing its performance on series of low dose frames will be a main goal of our future work.
Notes
Acknowledgements
Electron microscopy simulations and other work by ABY and PMV were supported by the US Department of Energy, Office of Basic Energy Sciences, Division of Materials Sciences and Engineering under Award #DEFG0208ER46547. PB and WD acknowledge funding from the National Science Foundation under Grant No. DMS1222390. The authors at RWTH Aachen were funded in part by the Excellence Initiative of the German Federal and State Governments.
References
 1.Batson, PE, Dellby, N, Krivanek OL: Subångstrom resolution using aberration corrected electron optics. Nature. 418(6898), 617–620 (2002).CrossRefGoogle Scholar
 2.Berkels, B, Binev, P, Blom, DA, Dahmen, W, Sharpley, R, Vogt, T: Optimized imaging using nonrigid registration. Ultramicroscopy. 138, 46–56 (2014).CrossRefGoogle Scholar
 3.Ortalan, V, Uzan, A, Gates, BC, Browning, ND: Direct imaging of single metal atoms and clusters in the pores of dealuminated HY zeolite. Nat. Nanotechnol. 5, 506–510 (2010).CrossRefGoogle Scholar
 4.Houben, L, Thust, A, Urban, K: Atomicprecision determination of the reconstruction of a 90 degree tilt boundary in YBa _{2} Cu _{3} O _{7−δ} by aberration corrected HRTEM. Ultramicroscopy. 106(3), 200–214 (2006).CrossRefGoogle Scholar
 5.Jia, CL, Urban, K, Alexe, M, Hesse, D, Vrejoiu, I: Direct observation of continuous electric dipole rotation in fluxclosure domains in ferroelectric Pb(Zr,Ti)O _{3}. Science. 331(6023), 1420–1423 (2011).CrossRefGoogle Scholar
 6.Schmid, H, Okunishi, E, Oikawa, T, Mader, W: Structural and elemental analysis of iron and indium doped zinc oxide by spectroscopic imaging in cscorrected. STEM Micron. 43(1), 49–56 (2012).CrossRefGoogle Scholar
 7.Kimoto, K, Asaka, T, Yu, X, Nagai, T, Matsui, Y, Ishizuka, K: Local crystal structure analysis with several picometer precision using scanning transmission electron microscopy. Ultramicroscopy. 110(7), 778–782 (2010).CrossRefGoogle Scholar
 8.Yankovich, AB, Berkels, B, Dahmen, W, Binev, P, Sanchez, SI, Bradley, SA, Li, A, Szlufarska, I, Voyles, PM: Picometreprecision analysis of scanning transmission electron microscopy images of platinum nanocatalysts. Nat. Commun (2014). doi:10.1038/ncomms5155.Google Scholar
 9.Nellist, PD, Pennycook, SJ: Accurate structure determination from image reconstruction in adf stem. J. Microsc. 190(12), 159–170 (1998).CrossRefGoogle Scholar
 10.Rosenauer, A, Mehrtensx, T, Müller, K, Gries, K, Schowalter, M, Venkata Satyam, P, Bley, S, Tessarek, C, Hommel, D, Sebald, K, Seyfried, M, Gutowski, J, Avramescu, A, Engl, K, Lutgen, S: Composition mapping in ingan by scanning transmission electron microscopy. Ultramicroscopy. 111(8), 1316–1327 (2011).CrossRefGoogle Scholar
 11.Kim, YM, He, J, Biegalski, MD, Ambaye, H, Lauter, V, Christen, HM, Pantelides, ST, Pennycook, SJ, Kalinin, SV, Borisevich, AY: Probing oxygen vacancy concentration and homogeneity in solidoxide fuelcell cathode materials on the subunitcell level. Nat. Mater. 11(10), 888–894 (2012).CrossRefGoogle Scholar
 12.Morgan, DG, Ramasse, QM, Browning, ND: Application of twodimensional crystallography and image processing to atomic resolution zcontrast images. J. Electron Microsc. 58(3), 223–244 (2009).CrossRefGoogle Scholar
 13.Unwin, PNT, Henderson, R: Molecular structure determination by electron microscopy of unstained crystalline specimens. J. Mol. Biol. 94(3), 425–440 (1975).CrossRefGoogle Scholar
 14.Klug, A: Image analysis and reconstruction in the electron microscopy of biological macromolecules. Chemica Scripta. 14, 1978–1979.Google Scholar
 15.Hovmöller, S, Sjögren, A, Farrants, G, Sundberg, M, Marinder, BO: Accurate atomic positions from electron microscopy. Nature. 311, 238–241 (1984).CrossRefGoogle Scholar
 16.Hovmöller, S: Crisp: crystallographic image processing on a personal computer. Ultramicroscopy. 41(1), 121–135 (1992).CrossRefGoogle Scholar
 17.Buades, A, Coll, B, Morel, JM: A review of image denoising algorithms, with a new one. Multiscale Model. Simul. 4(2), 490–530 (2005).CrossRefGoogle Scholar
 18.Binev, P, BlancoSilva, F, Blom, D, Dahmen, W, Lamby, P, Sharpley, R, Vogt, T: Modeling Nanoscale Imaging in Electron Microscopy. Nanostructure Science and Technology. In: Vogt, T, Dahmen, W, Binev, P (eds.), pp. 127–145. Springer, New York City (2012).Google Scholar
 19.Wei, DY, Yin, CC: An optimized locally adaptive nonlocal means denoising filter for cryoelectron microscopy data. J. Struct. Biol. 172(3), 211–218 (2010).CrossRefGoogle Scholar
 20.Mevenkamp, N, Yankovich, AB, Voyles, PM, Berkels, B: Nonlocal means for scanning transmission electron microscopy images and poisson noise based on adaptive periodic similarity search and patch regularization. In: Bender, J, Kuijper, A, von Landesberger, T, Theisel, H, Urban, P (eds.)VMV 2014: Vision, Modeling & Visualization, pp. 63–70. Eurographics Association, Darmstadt, Germany (2014).Google Scholar
 21.Bergmann, R: The fast Fourier transform and fast wavelet transform for patterns on the torus. Appl. Comput. Harmonic Anal. 35(1), 39–51 (2013).CrossRefGoogle Scholar
 22.Coupé, P, Yger, P, Prima, S, Hellier, P, Kervrann, C, Barillot, C: An Optimized Blockwise Nonlocal Means Denoising Filter for 3D Magnetic Resonance Images. IEEE Trans. Med. Imaging. 27(4), 425–441 (2008).CrossRefGoogle Scholar
 23.Salmon, J, Harmany, Z, Deledalle, CA, Willett, R: Poisson noise reduction with nonlocal PCA. J. Math. Imaging Vis. 48(2), 279–294 (2013).CrossRefGoogle Scholar
 24.Dabov, K, Foi, A, Katkovnik, V, Egiazarian, K: Image Denoising by sparse 3D transformdomain collaborative filtering. Image Process. IEEE Trans. 16(8), 2080–2095 (2007).CrossRefGoogle Scholar
 25.Anscombe, FJ: The transformation of poisson, binomial and negativebinomial data. Biometrika Trust. 35, 246–254 (1948).CrossRefGoogle Scholar
 26.Mäkitalo, M, Foi, A: Optimal inversion of the anscombe transformation in lowcount poisson image denoising. Image Process. IEEE Trans. 20(1), 99–109 (2011).CrossRefGoogle Scholar
 27.Deledalle, CA, Tupin, F, Denis, L: Poisson NL means: Unsupervised non local means for Poisson noise. In: Paper presented at the 17th Internal Conference on Image Processing, pp. 801–804. IEEE, Hong Kong (26–29 September 2010).Google Scholar
 28.Dabov, K, Foi, A, Katkovnik, V, Egiazarian, K: Image denoising with blockmatching and 3D filtering. Proc. SPIE. 6064, 606414–60641412 (2006).Google Scholar
 29.Maggioni, M, SánchezMonge, E, Foi, A, Danielyan, A, Dabov, K, Katkovnik, V, Egiazarian, K: Image and Video Denoising by Sparse 3D Transformdomain Collaborative Filtering: Blockmatching and 3D Filtering (BM3D) Algorithm and Its Extensions. Tampere University of Technology, http://www.cs.tut.fi/~foi/GCFBM3D/ Accessed date November, 10th 2014.
 30.Mäkitalo, M, Foi, A: Optimal Inversion of the Anscombe and Generalized Anscombe Variancestabilizing Transformations. Tampere University of Technology. http://www.cs.tut.fi/~foi/invansc/ Accessed date November, 10th 2014.
 31.Martin, DC, Chen, J, Yang, J, Drummy, LF, Kübel, C: High resolution electron microscopy of ordered polymers and organic molecular crystals: Recent developments and future possibilities. J. Polymer Sci. Part B: Polymer Phys. 43(14), 1749–1778 (2005).CrossRefGoogle Scholar
 32.Kirkland, EJ: Advanced Computing in Electron Microscopy. 2nd edn. Springer, New York City (2010).CrossRefGoogle Scholar
 33.Van Aert, S, Den Dekker, AJ, Van Den Bos, A, Van Dyck, D, Chen, JH: Maximum likelihood estimation of structure parameters from high resolution electron microscopy images. Part II: a practical example. Ultramicroscopy. 104(2), 107–125 (2005).CrossRefGoogle Scholar
Copyright information
This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited.