Introduction

Super-resolution (SR) fluorescence microscopy is a powerful tool to obtain higher spatial resolution in biological imaging. Many techniques, such as photoactivated localization microscopy (PALM) (Betzig et al. 2006; Hess et al. 2006), stochastic optical reconstruction microscopy (STORM) (Huang et al. 2008; Rust et al. 2006), stimulated emission depletion (STED) (Klar and Hell 1999), super-resolution optical fluctuation imaging (SOFI) (Dertinger et al. 2009), and structured illumination microscopy (SIM) (Gustafsson 2000; Gustafsson et al. 2008), have been proposed. Among current techniques, SIM is often used in the living cell imaging process because of its fast speed and need of illumination of low intensity. Compared with STED, PALM/STORM methods that focus on modulation in the spatial domain, SIM manipulates in the frequency domain to boost resolution (Dan et al. 2014). Thus, the required sample preparation and imaging process are the same as for wide-field fluorescence microscopy. Consequently, any fluorescent sample imaged using wide-field microscopy is compatible with SIM. Moreover, the temporal resolution is as important as the spatial resolution in living cell studies. Many SR techniques attain high spatial resolution by sacrificing the temporal resolution, such as PALM/STORM and SOFI (Kapanidis and Strick 2009). On the other hand, SIM is a notable for its fast speed in imaging live cells (Kner et al. 2009).

Because of these advantages, SIM has been broadly combined with many imaging modes, such as total-internal reflection fluorescence microscopy (TIRF–SIM) (Li et al. 2015), light sheet microscopy (Chen et al. 2014), and surface plasmon microscopy (Wei and Liu 2010). There are also a series of developments in atypical SIM, for example, image scanning microscopy (ISM) (Schulz et al. 2013) and SIM with speckle pattern (blind-SIM) (Mudry et al. 2012). To reduce the artifacts in the reconstructed results, there have been many developments in SIM microscopy, including the strategic and practical protocol of sample preparation and system calibration (Demmerle et al. 2017), more accurate parameter estimation methods for SIM (Chu et al. 2014; Huang et al. 2018; Shroff et al. 2009; Wicker 2013; Wicker et al. 2013; Zhou et al. 2016), SIM reconstruction based on Bayesian estimation (Orieux et al. 2012) or with Richardson–Lucy deconvolution method for Poisson noise (Perez et al. 2016), and some open-source reconstruction programs (Křížek et al. 2016; Lal et al. 2016; Müller et al. 2016). Iterative reconstruction methods have also been proposed to achieve better artifact-suppression during the reconstruction of low SNR data (Chu et al. 2014; Huang et al. 2018).

By restricting the excitation to only a small fraction of the cellular volume (Li et al. 2015), TIRF–SIM achieves SR with reduced phototoxicity and out-of-focus background. However, the faster imaging rate and shorter exposure time render the signal-to-noise ratio of raw TIRF–SIM data to become extremely low. In addition, low-dose or low-efficiency fluorochromes can also be an important cause of a low signal-to-noise ratio. We find that parameter estimation and Wiener deconvolution, which are commonly used, cannot work well in low signal-to-noise ratio conditions (Gustafsson et al. 2008; Lal et al. 2016; Müller et al. 2016). Furthermore, uneven field illumination and low modulation may also affect the reconstruction of raw data. During the data reconstruction process, researchers, especially novice users, may not recognize these factors that may introduce magnified artifacts. Thus, we need to check the quality of raw data and use a modified parameter estimation and deconvolution method during the reconstruction process to reduce artifacts in the reconstruction results.

We used SIMcheck to judge the quality of raw data and employed an effective algorithm for parameter estimation and deconvolution that was specifically developed for the low SNR condition (Ball et al. 2015). Then, we listed the advantages of each method in the following section and proposed a full reconstruction protocol for reconstruction under the low SNR condition. Compared with the commonly used method, this protocol can more efficiently minimize the artifacts in reconstructed SR images.

Summarized procedure

  1. (1)

    Capture the ultrafast SIM raw data.

  2. (2)

    Compute the MIV and MCNR value of raw images with the SIMcheck software in ImageJ plugin [TIMING] ~10 s

    [CRITICAL STEP] The sample may have a fast movement or an uneven illumination pattern, and a non-white area in MIV result could indicate the precise location of these flaw, which will cause serious artifacts in the reconstructed SR images. These areas of reconstructed SR images should be recognized as invalid areas corresponding to the non-white areas in MIV result. The value of MCNR could determine the deconvolution method in the reconstruction process.

  3. (3)

    Estimate the parameters with a modified method [TIMING] ~60 s

    A modified method could guarantee the precision of parameter estimation even the SNR of SIM raw data is extremely low.

  4. (4)

    A deconvolution method was chosen corresponding to the MCNR value of SIMcheck.

    [CRITICAL STEP] If the value exceeded 4, the Wiener filter in Fair-SIM or RL deconvolution is sufficient. In contrast, the iterative method (TV and Hessian) should be considered if the MCNR is below 4.

Theory and reconstruction for TIRF–SIM

Image theory

Structured illumination microscopy (SIM) is a wide-field SR imaging method that measures the details in the frequency domain, using the Moiré pattern (Gustafsson 2000). The patterns transform the high-frequency components to low-frequency information during the exposure process. The patterns can be described as a sinusoidal function and are discretized by a spatial light modulator (SLM). Iθ,φ is the sinusoidal intensity pattern for illumination and is described by the following equation:

$$I_{\theta ,\varphi } \left( {\varvec{r}} \right) = I_{0} \left[ {1 - c_{\theta } \cdot \cos \left( {2\uppi {\varvec{p}}_{\theta } \cdot {\varvec{r}} + \varphi_{\theta } } \right)} \right],$$
(1)

where r ≡ (x, y) is the two-dimensional space position vector, I0 is the illumination intensity, cθ is the modulation depth, φθ is the starting phase, and pθ represents the pattern wave vector in the illumination pattern orientation θ. Let g(r) represents the fluorescence density distribution of the sample; then, the observed emission distribution Dθ,φ(r) can be described as follows:

$$D_{\theta ,\varphi } ({\varvec{r}}) = \left[ {g({\varvec{r}})I_{\theta ,\varphi } ({\varvec{r}})} \right] \otimes h({\varvec{r}}),$$
(2)

where h(r) is the point spread function of the SIM set-up and ⊗  is the convolution operator. The Fourier transform of the observed raw data distribution Dθ,φ(r) is given by the following equation:

$$D_{\theta ,\varphi } ({\varvec{k}}) = [g({\varvec{k}}) \otimes I_{\theta ,\varphi } ({\varvec{k}})] \cdot H({\varvec{k}}),$$
(3)

where H(k) is the optical transfer function (OTF) and Iθ,φ(k) is the Fourier transform of the illumination sinusoidal intensity pattern in Eq. 1. Dθ,φ(k) can be represented as follows:

$$D_{\theta ,\varphi } ({\varvec{k}}) = I_{0} \left[ {g({\varvec{k}}) - \frac{{c_{\theta } }}{2}g({\varvec{k}} - {\varvec{p}}_{\theta } ){\text{e}}^{{ - i\varphi_{\theta } }} - \frac{{c_{\theta } }}{2}g({\varvec{k}} + {\varvec{p}}_{\theta } ){\text{e}}^{{ + i\varphi_{\theta } }} } \right] \times H({\varvec{k}}),$$
(4)

where g(k \(\pm\,\varvec{p}_{\theta }\)) is the \(\pm \varvec{p}_{\theta }\) frequency shift of g(k) and contains information components that are normally outside of the OTF H(k). To separate g(k \(+\,\varvec{p}_{\theta }\)), g(k \(-\,\varvec{p}_{\theta }\)), and g(k), three different phases, φ (1) θ , φ (2) θ , and φ (3) θ , respectively, are applied to the illumination pattern as substitutes for φθ in Eq. 1. Then, the raw data distributions \(D_{{\theta ,\varphi^{\left( 1 \right)} }} ({\varvec{k}})\), \(D_{{\theta ,\varphi^{\left( 2 \right)} }} ({\varvec{k}})\), and \(D_{{\theta ,\varphi^{\left( 3 \right)} }} ({\varvec{k}})\) are obtained, respectively, with three different phases, φ (1) θ , φ (2) θ , and φ (3) θ , with the illumination pattern orientation θ. The phases φ (1) θ , φ (2) θ , and φ (3) θ are unknown; however, the phase differences φ (2) θ  − φ (1) θ  = 120° and φ (3) θ  − φ (2) θ  = 120° are determined in a typical case, and then the different phases in one orientation can be described as follows:

$$\left[ {\begin{array}{*{20}c} {D_{{\theta ,\varphi^{\left( 1 \right)} }} \left( {\varvec{k}} \right)} \\ {D_{{\theta ,\varphi^{\left( 2 \right)} }} \left( {\varvec{k}} \right)} \\ {D_{{\theta ,\varphi^{\left( 3 \right)} }} \left( {\varvec{k}} \right)} \\ \end{array} } \right] = I_{0} {\mathbf{M}}\left[ {\begin{array}{*{20}c} {g\left( {\varvec{k}} \right) \cdot H\left( {\varvec{k}} \right)} \\ {g\left( {{\varvec{k}} - {\varvec{p}}_{\theta } } \right) \cdot H\left( {\varvec{k}} \right)} \\ {g\left( {{\varvec{k}} + {\varvec{p}}_{\theta } } \right) \cdot H\left( {\varvec{k}} \right)} \\ \end{array} } \right] ,$$
(5)

where the matrix M is

$${\varvec{M}} = \left[ {\begin{array}{*{20}c} 1 & { - \frac{{c_{\theta } }}{2}{\text{e}}^{{ - i\varphi_{\theta }^{\left( 1 \right)} }} } & { - \frac{{c_{\theta } }}{2}{\text{e}}^{{ + i\varphi_{\theta }^{\left( 1 \right)} }} } \\ 1 & { - \frac{{c_{\theta } }}{2}{\text{e}}^{{ - i\varphi_{\theta }^{\left( 2 \right)} }} } & { - \frac{{c_{\theta } }}{2}{\text{e}}^{{{ + }i\varphi_{\theta }^{\left( 2 \right)} }} } \\ 1 & { - \frac{{c_{\theta } }}{2}{\text{e}}^{{ - i\varphi_{\theta }^{\left( 3 \right)} }} } & { - \frac{{c_{\theta } }}{2}{\text{e}}^{{{ + }i\varphi_{\theta }^{\left( 3 \right)} }} } \\ \end{array} } \right] .$$
(6)

After the sample data distributions \(D_{{\theta_{1} ,\varphi^{\left( 1 \right)} }} ({\varvec{k}})\), \(D_{{\theta_{1} ,\varphi^{\left( 2 \right)} }} \left( {\varvec{k}} \right)\), and \(D_{{\theta_{1} ,\varphi^{\left( 3 \right)} }} \left( {\varvec{k}} \right)\) from the image are obtained in the first orientation θ1, the process is repeated twice for two illumination orientations, θ2 and θ3, to cover the whole SR-frequency domain. The nine frames of raw TIRF–SIM data are gathered after the repetition of the three orientations and will be used together to reconstruct one frame of an SIM SR image via the following procedure.

Reconstruction

The conventional procedure for reconstruction can be divided into two important steps. First, we estimate the three unknown parameter values: the pattern wave vector pθ, the start phase φ (1) θ , and the modulation depth cθ. Second, the raw data are reconstructed with these parameters via the Wiener filter.

Parameter estimation

In the first step, we use the three raw data distributions: \(D_{{\theta_{1} ,\varphi^{\left( 1 \right)} }} \left( {\varvec{k}} \right)\), \(D_{{\theta_{1} ,\varphi^{\left( 2 \right)} }} \left( {\varvec{k}} \right)\) , and \(D_{{\theta_{1} ,\varphi^{\left( 3 \right)} }} \left( {\varvec{k}} \right)\) that have the same orientation θ1. These three raw data distributions are a linear combination of the different frequency distributions g(k) and g(k \(\pm\,\varvec{p}_{{\theta_{1} }}\)), including the low-pass optical transfer function H(k). Taking the phase differences φ (2) θ  − φ (1) θ  = 120° and φ (3) θ  − φ (2) θ  = 120°, Eq. 5 can be rewritten as follows:

$$\left[ {\begin{array}{*{20}c} {D_{{\theta_{1} ,\varphi^{\left( 1 \right)} }} \left( {\varvec{k}} \right)} \\ {D_{{\theta_{1} ,\varphi^{\left( 2 \right)} }} \left( {\varvec{k}} \right)} \\ {D_{{\theta_{1} ,\varphi^{\left( 3 \right)} }} \left( {\varvec{k}} \right)} \\ \end{array} } \right] = I_{0} \left[ {\begin{array}{*{20}c} 1 & {{\text{e}}^{ - 0i} } & {{\text{e}}^{0i} } \\ 1 & {{\text{e}}^{{ - \frac{2{\pi} i}{3}}} } & {{\text{e}}^{{\frac{2\pi i}{3}}} } \\ 1 & {{\text{e}}^{{ - \frac{4{\pi} i}{3}}} } & {{\text{e}}^{{\frac{4{\pi} i}{3}}} } \\ \end{array} } \right]\left[ {\begin{array}{*{20}c} {g\left( {\varvec{k}} \right) \cdot H\left( {\varvec{k}} \right)} \\ {g\left( {{\varvec{k}} - {\varvec{p}}_{{\theta_{1} }} } \right) \cdot H\left( {\varvec{k}} \right) \cdot \frac{{ - c_{{\theta_{1} }} }}{2}{\text{e}}^{{ - i\varphi_{{\theta_{1} }}^{\left( 1 \right)} }} } \\ {g\left( {{\varvec{k}} + {\varvec{p}}_{{\theta_{1} }} } \right) \cdot H\left( {\varvec{k}} \right) \cdot \frac{{ - c_{{\theta_{1} }} }}{2}{\text{e}}^{{ + i\varphi_{{\theta_{1} }}^{\left( 1 \right)} }} } \\ \end{array} } \right].$$
(7)

The frequency-shifted component \(g\left( {{\varvec{k}} \pm {\varvec{p}}_{{\theta_{1} }} } \right)\) is the high-frequency counterpart of g(k), which means \(g\left( {{\varvec{k}} \pm {\varvec{p}}_{{\theta_{1} }} } \right)\) has common information in the sections that overlap with g(k). We can calculate the cross-correlation of the overlap sections to estimate the pattern wave vector \({\varvec{p}}_{{\theta_{1} }}\). The cross-correlation of \({\varvec{C}}_{1} \left( {\varvec{k}} \right) = g\left( {\varvec{k}} \right) \cdot H\left( {\varvec{k}} \right) \cdot H\left( {{\varvec{k}} + {\varvec{p^{\prime}}}_{{\theta_{1} }} } \right)\) with the \({\varvec{p^{\prime}}}_{{\theta_{1} }}\) shifted variant \({\mathbf{C}}_{2} \left( {{\varvec{k}} + {\varvec{p^{\prime}}}_{{\theta_{1} }} } \right) = g\left( {{\varvec{k}} + {\varvec{p^{\prime}}}_{{\theta_{1} }} - {\varvec{p}}_{{\theta_{1} }} } \right) \cdot H\left( {{\varvec{k}} + {\varvec{p^{\prime}}}_{{\theta_{1} }} } \right) \cdot H\left( {\varvec{k}} \right) \cdot \frac{{ - c_{{\theta_{1} }} }}{2}{\text{e}}^{{ - i\varphi_{{\theta_{1} }}^{\left( 1 \right)} }}\) is the maximum when \({\varvec{p^{\prime}}}_{{\theta_{1} }}\) equals \({\varvec{p}}_{{\theta_{1} }}\) at any given modulation depth cθ and starting phase \(\varphi_{{\theta_{1} }}^{\left( 1 \right)}\).

After a subpixel accuracy estimation of the pattern wave vector \({\varvec{p}}_{{\theta_{1} }}\), the starting phase \(\varphi_{{\theta_{1} }}^{\left( 1 \right)}\) and modulation depths \(c_{{\theta_{1} }}\) are estimated by linear regression of the overlap at different coordinate k in the frequency domain. Similarly, the parameter estimation process for the other two orientations, θ2 and θ3, is the same as for the orientation θ1.

Reconstruction with the Wiener filter

Once the parameters are estimated, we can separate the nine SR-frequency components iθ,m from the known parameters \(\varphi_{{\theta_{1} }}^{\left( 1 \right)}\) and \(c_{{\theta_{1} }}\):

$$\left[ {\begin{array}{*{20}c} {i_{{\theta ,m_{0} }} } \\ {i_{{\theta ,m_{ - 1} }} } \\ {i_{{\theta ,m_{ + 1} }} } \\ \end{array} } \right] = I_{0} \cdot \left[ {\begin{array}{*{20}c} {g\left( {\varvec{k}} \right) \cdot H\left( {\varvec{k}} \right)} \\ {g\left( {{\varvec{k}} - {\varvec{p}}_{\theta } } \right) \cdot H\left( {\varvec{k}} \right)} \\ {g\left( {{\varvec{k}} + {\varvec{p}}_{\theta } } \right) \cdot H\left( {\varvec{k}} \right)} \\ \end{array} } \right],$$
(8)

where mth= −1, 0, +1 denotes the order of the bands. Then, we shift iθ,m to the proper place Sθ,m with the known wave vector \({\varvec{p^{\prime}}}_{{\theta_{1} }}\):

$$\left[ {\begin{array}{*{20}c} {S_{{\theta ,m_{0} }} } \\ {S_{{\theta ,m_{ - 1} }} } \\ {S_{{\theta ,m_{ + 1} }} } \\ \end{array} } \right] = I_{0} \cdot \left[ {\begin{array}{*{20}c} {g\left( {\varvec{k}} \right) \cdot H\left( {\varvec{k}} \right)} \\ {g\left( {\varvec{k}} \right) \cdot H\left( {{\varvec{k}} + {\varvec{p}}_{\theta } } \right)} \\ {g\left( {\varvec{k}} \right) \cdot H\left( {{\mathbf{k}} - {\varvec{p}}_{\theta } } \right)} \\ \end{array} } \right].$$
(9)

In the above equation, Sθ,m is the frequency band of the shifted components corresponding to the θ direction and the mth order, θ = 1–3 is the pattern orientation, and mth= −1, 0, +1. The Wiener filter can combine the different components into a SR image through a weighted average process during the deconvolution, as indicated in the following equation:

$$g\left( r \right) = ifft\left( {\frac{{\sum\nolimits_{m,\theta }^{{}} {H^{*} \left( {{\varvec{k}} + m{\varvec{p}}_{\theta } } \right)S_{\theta ,m} } }}{{\sum\nolimits_{m,\theta }^{{}} {\left| {H\left( {{\varvec{k}} + m{\varvec{p}}_{\theta } } \right)} \right|^{2} } + \alpha^{2} }}A\left( {\varvec{k}} \right)} \right).$$
(10)

In the above equation, H is the optical transfer function, * is the conjugate transpose, k is the frequency domain coordinates, pθ is the pattern wave vector of the θ pattern orientation, α is the Wiener parameter, and A(k) is an apodization function used to eliminate artifacts (Gustafsson et al. 2008).

Ultrafast SIM

Artifact analysis

The reconstruction results for ultrafast structured illumination microscopy (ultrafast SIM) are disturbed by artifacts that seriously affected the data analysis. Thus, the causes of these artifacts need to be analyzed and solved individually. In summary, the artifacts originate mainly from two sources: optical imperfection and noise disturbance in the imaging process.

Optical imperfection causes a regular artifact that can be found in the reconstructed images. However, these artifacts can confuse new users of SIM microscopy. SIMcheck, a plugin for ImageJ, was developed to check a series of important criteria of the raw data and reconstruction results to find the origins of these artifacts (Ball et al. 2015).

Noise includes readout noise, dark current and shot noise. These types of noise can be regarded as white noise, which is frequency dependent in the frequency domain and unique random in the time domain. These kinds of noise lead to the raw data being distinctly disturbed with a mix of Poisson and Gaussian noise, especially for ultrafast exposure times.

This mixing noise in the spatial domain will render precise estimations of the pattern wave vector more difficult during the reconstruction of ultrafast SIM data. If the pattern wave vector \({\varvec{p}}_{{\theta_{1} }}\) is not estimated accurately, the subsequent estimation of the starting phase φmd and the modulation depth c will deviate severely due to the error \({\varvec{p}}_{{\theta_{1} }}\). Then, the 0 and ±1 spatial frequency components will not be perfectly separated and will be shifted to an incorrect location with an error vector \({\varvec{p}}_{{\theta_{1} }}\) in Fourier space, leading to regular artifacts in the reconstructed SR images. These artifacts, which are caused by the deviation of the initial parameter determination, are obviously distinguished in the SIM reconstructed images. An error in the pattern wave vectors will lead to a patterned artifact, such as a snowflake shape, an error in the initial phase will lead to an asymmetric ring artifact, and an error in the modulation contrast will lead to a change in the smoothness or shape of the high-frequency signal (Huang et al. 2018). In addition, this mixing noise will be amplified by the classic Wiener deconvolution method. Thus, the reconstructed SR images have disturbing artifacts that obscure the structure of biological sample.

Motion artifacts or uneven patterns

Before reconstructing raw images of ultrafast SIM, we propose that the following pivotal criteria provided by the SIMcheck software should be checked. A motion and illumination variation (MIV) feature in SIMcheck obtains normalized intensity of the phase-shifted images acquired for each plane for the different illumination patterns and highlights uneven field illumination or movement during acquisition with a colored subregion map. Modulation contrast is another important parameter of the raw data quality, as it critically affects the amount of the frequency-shifted information that can be reassigned during the reconstruction process. The process of calculating the modulation contrast-to-noise (MCN) ratio within the image and translating this information to a heat map provides an average modulation contrast-to-noise ratio (MCNR) as a criterion for evaluating image quality.

Modified parameter estimation

Under the high-NA TIRF–SIM configuration, overlapping regions of different spectra decrease. Furthermore, even fewer excited photons are captured per exposure during the ultrafast exposure time. The low SNR raw data in the spatial domain will render precise estimations of the parameters more difficult; thus, the modified parameter estimation method needs to be considered. The filter method and the averaged method can be simply applied in the conventional reconstruction process under this low SNR condition (Chu et al. 2014; Huang et al. 2018). In addition, iterative and non-iterative methods have been proposed to estimate the parameters even when the pattern ratio in an experimental setup is unknown (Wicker 2013; Wicker et al. 2013). An image recombination transform algorithm can obtain the initial phase of fringe with high precision even with high background and weak modulation depth (Zhou et al. 2016).

Reconstruction software

A number of software exists for users to reconstruct SR images. Fair-SIM, an open-source plugin for ImageJ, contains extensive functions for the reconstruction process. This plugin can process the TIRF–SIM, two-dimensional SIM and three-dimensional SIM data captured from a homemade setup or a commercial SIM microscope, such as OMX and Zeiss. With the parameter determination approximated by the OTF, Fair-SIM can estimate the parameters \({\varvec{p}}_{{\theta_{1} }}\), \(\varphi_{{\theta_{1} }}^{\left( 1 \right)}\) and \(c_{{\theta_{1} }}\) rapidly. Then, SR images can be reconstructed continuously with the Wiener deconvolution or Richardson Lucy deconvolution depending on the noise level.

A two-filter step with Richardson Lucy deconvolution (RL deconvolution) method exists for effective parameter estimation (Perez et al. 2016). If the raw data are corrupted with Poisson noise or an out-of-focus background, this two-filter steps method can effectively move a fluctuating reconstruction spectrum artifact caused by an out-of-focus background or a discontinuous artifact caused by Poisson noise.

The iterative method has also been applied in the reconstruction of SR images, such as TV and Hessian deconvolution (Chu et al. 2014). These iterative methods were based on Gaussian noise and used a priori knowledge, such as the continuity of the biological sample and the PSF, to establish an optimization function. These iterative methods can restrain the artifacts more effectively than the above methods but require more computation.

A protocol for parameter estimation and reconstruction

We proposed a procedure to choose a suitable reconstruction method. First, we used SIMcheck to compute the MCNR and MIV values. A non-white MIV area indicates possible blurring of the SR images due to motion artifacts or an uneven pattern, and this error was resolved by increasing the imaging frame rate or checking the illumination pattern. Then, a modified parameter estimation method was used to improve the estimation precision (refer to the section of “Modified parameter estimation”). Finally, a deconvolution method corresponding to the MCNR value of SIMcheck was chosen. If the value exceeded 4, the Wiener filter in Fair-SIM or RL deconvolution is sufficient. In contrast, the iterative method (TV and Hessian) should be considered if the MCNR is below 4. The TV deconvolution method is appropriate for a flaky structure, and the Hessian–SIM method is applied for situations with flakiness or a line structure. Furthermore, if the biological sample moves slower than 65 nm per frame, time-axis continuity can be used in the Hessian method to better minimize the artifacts in the reconstruction (Huang et al. 2018).

Experiments

We tested our reconstruction protocol on a synthetic sample that combined different structures and low signal-to-noise ratio actin with a 0.5-ms exposure time.

The synthetic sample used for the simulations is shown in Fig. 1A, with a maximum intensity of 109 a.u.. To imitate the actual exposure process, the phantom was illuminated with the pattern presented in Eq. 1. The simulated excitation wavelength λex was 488 nm, the emission wavelength λem was 515 nm, and the numerical aperture (NA) was 1.7. We then added Poisson distributed noise to the illuminated images. We also added a background value of 99 a.u. to all pixels, in combination with Gaussian noise with a standard deviation from 1 to 20 a.u. At the same time, we also considered that there may be fast movements or an uneven illumination pattern, which will cause serious artifacts in the reconstructed SR images, during the raw data-capture process. We illuminated a shifting structure with three different pattern phases and orientations to simulate a rapid movement during the SIM raw data-capture process. We then followed the above protocol with the SIMcheck software to calculate the MIV of the SIM raw images and the reconstruction of SR images from the simulation data.

Fig. 1
figure 1

The non-white MIV and motion artifacts caused by a fast-moving ring structure. A First frame of the SIM raw data before pattern illumination. B Magnified region from the nine-frame raw image. The ROI of the nine frames are different from each other to simulate a fast-moving biological structure, and a yellow box was plotted at the same position to highlight the difference between the upper left and the low right images. C MIV result from the nine-frame raw data. The non-white area represents the motion artifacts in the reconstructed SR images. D Magnified view of the reconstructed SR images. Motion artifacts are evident around the fast-moving structures in D, and the MIV result is also non-white in the same area shown in C. Scale bar, 2 µm

Full size image

In addition, we also applied two different parameter estimation methods to the synthetic images of different noise levels to evaluate the bias of each method compared with the ground truth. Then, we used the non-iterative Wiener method provided by Fair-SIM and the Hessian method separately to reconstruct the noisy synthetic image. Finally, we used the following evaluation criteria to judge the quality of the reconstruction images.

The biological sample actin was exposed for 0.5 ms, and data were taken continually seven times before the subsequent phase, and orientation information were collected. We averaged the seven frames with the same phase and orientation into one frame, which had a higher SNR and could be regarded as the ground truth. We also extracted one frame from every seven frames to represent ultrafast raw data with a 0.5-ms exposure time. The excitation wavelength λex was 488 nm, the emission wavelength λem was 515 nm, and the numerical aperture NA was 1.7. We applied the conventional reconstruction process and the proposed protocol to these raw data and compared the artifacts of the different reconstruction methods with the following evaluation criteria.

Evaluation criteria

To compare the performances of different methods, we adopted the following criteria: the peak signal-to-noise ratio (PSNR), and the structural similarity (SSIM) (Wang et al. 2004).

PSNR is defined as

$${PSNR} = 10\log_{10} \left( {\frac{{\mu_{\hbox{max} }^{2} }}{MSE}} \right),$$
(11)

where μmax is the maximum possible pixel value of the image and MSE is the mean-squared error between the reconstructed image and the original image. A higher PSNR means less difference between the reconstructed image and the original image.

SSIM is defined as

$${SSIM}\, (a,b )= \frac{{\left( {2\mu_{a} \mu_{b} + C_{1} } \right)\left( {2\sigma_{ab} + C_{2} } \right)}}{{\left( {\mu_{a}^{2} + \mu_{b}^{2} + C_{1} } \right)\left( {\sigma_{a}^{2} + \sigma_{b}^{2} + C_{2} } \right)}},$$
(12)

where a and b are two windows of size 11 × 11 pixels in the same position of two images. μa and μb are the averages of window a and b, respectively. σ 2 a and σ 2 b are the variances of window a and b, respectively. σxy is the covariance between the two windows. C1 and C2 were chosen as C1 = (0.01μmax)2 and C2 = (0.03μmax)2, respectively. A higher SSIM means that the region of these two images is more similar.

Simulation experiment

For reconstructions of the SR images, the results will contain a large number of artifacts scattered randomly throughout the SR images. Under the high signal-to-noise ratio condition, the signal of reconstructed SR images is relatively strong, and the artifacts are not obvious in the reconstructed SR images. However, when the signal-to-noise ratio is reduced, the signal is relatively weaker, and the artifacts are more pronounced. The continuous signals of the reconstructed SR images become discontinuous because of the artifacts, which is the primary problem that we designed this protocol to solve. Therefore, we simulated data obtained with a very low photon number and applied Poisson and Gaussian noises with different standard deviations, which represented the ultrafast SIM raw data acquired with different exposure times. In addition, there are other several underlying problems, such as an uneven illumination pattern or fast movement of the biological structure during the acquisition process, which lead to severe artifacts in reconstructed SR images.

We first used the structured pattern to illuminate nine different images to simulate the capture process of fast-moving biological samples. Then, we use SIMcheck to calculate the MIV result of the raw images before reconstructing the SIM SR images. If the color of whole MIV result was white, we confirmed that there was no problem that may cause serious artifacts in the SIM raw images. Otherwise, a non-white area of MIV means that motion artifacts exist in the reconstructed SR images (Fig. 1).

By comparing the nine-frame raw data in Fig. 1B, we can see that the reconstructed SR image will produce motion artifacts (Fig. 1D) when there are fast-moving structures in the nine frames of SIM raw data. Additionally, the MIV result will also produce a non-white area at the corresponding position (Fig. 1C). Therefore, SIMcheck is a necessary step to pre-check SIM raw data before reconstructing SR images.

After pre-checking the raw data, the next procedure is to estimate the pattern wave vector p, the starting phase \(\varphi_{{\theta_{1} }}^{\left( 1 \right)}\) and the modulation depths \(c_{{\theta_{1} }}\) from the SIM raw data. We first estimated the pattern wave vectors of these series of simulation data with different noise levels using the Fair-SIM software. However, the conventional method had errors in the estimated results when the standard deviation was larger than 10 (the third column in Table 1). Thus, we used the modified parameter estimation method by averaging the raw data corresponding to their patterns and phases (Huang et al. 2018) and then estimated the pattern wave vectors of the averaged raw data using the Fair-SIM software (the fourth column in Table 1).

Table 1 Comparison of estimated pattern wave vectors by different estimation methods with different Gaussian noises
Full size table

As we can see from the table, a negligible error (0.14%) in the pattern wave vector was obtained with the conventional parameter estimation method when the standard deviation was below than 10, which could hardly generate artifacts caused by the pattern wave vector error (Huang et al. 2018). However, as the standard deviation of Gaussian noise increased (simulation of the lower SNR of raw images with shorter exposure times), the conventional parameter estimation method differed more from the ground truth parameter. When the standard deviation of Gaussian noise was greater than 10, an error of 46 pixels was generated compared with the ground truth. This error in the pattern wave vector can generate a serious error in the estimated starting phase \(\varphi_{{\theta_{1} }}^{\left( 1 \right)}\) and modulation depths \(c_{{\theta_{1} }}\), which could produce periodic artifacts in the reconstructed SR images. These artifacts were much more significant and severe because these SR images contained not only artifacts generated by noise but also artifacts due to the deviations of the parameters from the ground truth. We can see that artifacts due to errors in parameter estimation have a great impact on the reconstructed SR images, even completely obscuring the structures and details in the reconstructed SR image (Fig. 2).

Fig. 2
figure 2

Reconstructed SR images with different protocols and corresponding criteria compared with the ground truth. A Ground truth. B Reconstruction via erroneous parameters and the Wiener deconvolution method. C Reconstruction via the correct parameters and the Wiener deconvolution method. D Reconstruction via the correct parameters and the Hessian reconstruction method. E PSNR and SSIM criteria of SR images by three different methods compared with the ground truth (n = 8). Scale bar, 2 µm

Full size image

In contrast, the pattern wave vectors estimated using the average method (the fourth column in Table 1) always had a miniscule deviation (0.02%) from the ground truth (the second column in Table 1), even when the standard deviation of Gaussian noise was 20 (not shown in Table 1).

Under ultrashort exposures, the noise source of raw images may be dominated by Poisson but not Gaussian noise. Thus, we simulated raw images that had max pixel value of 29–44 a.u. and were corrupted with Poisson noise. We used the Fair-SIM to estimate this series of noisy raw data with custom and averaged method, and the results are presented as shown in Table 2.

Table 2 Comparison of estimated pattern wave vectors by different estimation methods with Poisson noise
Full size table

The pattern wave vectors estimated using the average method (the fourth column in Table 2) always had a miniscule deviation (0.02%) from the ground truth (the second column in Table 2). As the pixel value decreased (simulation of the raw images with shorter exposure time), the conventional parameter estimation method (the third column in Table 2) differed more from the ground truth parameter (the second column in Table 2), when the Max pixel value of raw data was smaller than 34, 50 pixels of error was generated compared with the ground truth. Notably, the estimation results of the third and fourth columns of Tables 1 and 2 were all calculated by the same algorithm provided by the Fair-SIM software, and the only difference was that the modified method simply averages the raw data according to the phase and direction of the pattern wave vectors. The averaged parameter estimation method only needs to average the SIM raw images according to their pattern and phase along the time axis; accurate results could then be obtained even with the same conventional parameter estimation method. The average method is simple and effective and has huge promising potential for combination with other algorithms (Chu et al. 2014; Shroff et al. 2009; Wicker 2013; Wicker et al. 2013; Zhou et al. 2016).

To distinguish the artifacts caused by the erroneous parameters from artifacts caused by the Wiener deconvolution, we used simulation data with Gaussian noise with a standard deviation of 2, which could cause a clear difference between these two artifacts in the reconstructed SR images. We artificially changed the pattern wave vector to a 2% error from the ground truth and then used the following three methods to reconstruct the low SNR raw data: (1) an error pattern wave vector and the Wiener deconvolution method; (2) modified parameter estimation and the Wiener deconvolution method; and (3) modified parameter estimation and the iterative Hessian algorithm. Then, the reconstructed SR images reconstructed by these three methods were compared with the ground-truth image using the above evaluation criteria.

As seen in Fig. 2, the Wiener method reconstructed SR image with an error parameter has the most significant artifacts and the lowest evaluation criteria compared with the ground truth (Fig. 2B). The reason is that SR image reconstructed by this method contains not only artifacts generated by noise but also artifacts due to the deviations in the parameter estimation. Comparing Fig. 2B with 2C, we found that artifacts due to parameter estimation errors have a greater impact on the reconstructed SR images. We cannot completely obscure the structures and details in the reconstructed SR image. Furthermore, we also found that artifacts were also generated in the SR images even with a small error in the parameter estimation, which came from the amplification of the noise by the Wiener deconvolution (Fig. 2C). As shown in the yellow box in Fig. 2C, continuous lines in the ground truth may also be intermittent due to the existence of artifacts. Therefore, an improved parameter estimation method is needed to obtain the correct parameters during the reconstruction, and modified deconvolution algorithms are needed to achieve a stronger suppression effect of artifacts, which is shown in Fig. 2D with the use of the iterative Hessian algorithm. The evaluation criteria (Fig. 2E) also indicated the proposed protocol attained the highest value compared with that of the other two methods.

Ultrafast exposed actin with SIM

The number of photons received by the sCMOS camera was further reduced in the ultrafast exposed raw data of actin. Under this condition, the intensity of the noise will affect the structured illumination pattern and actin structure in the SIM raw data. When estimating the parameters of SIM raw data using conventional methods and performing reconstruction with the Wiener deconvolution, severe artifacts are generated in the reconstructed SR results. We then used the following two protocols to reconstruct this low SNR biological sample: (1) the Wiener deconvolution method with the averaged parameter estimation method and (2) the modified parameter estimation and iterative Hessian reconstruction algorithm.

As shown in Fig. 3, even with the correct parameter estimation, with the averaged method, artifacts are also generated in the reconstructed SR images because of the amplification of noise by the Wiener deconvolution (Fig. 3B). Therefore, an improved parameter estimation method is needed to obtain the correct parameters during the reconstruction, and better deconvolution algorithms to suppress artifacts are needed. The use of iterative Hessian algorithms can better reduce artifacts, as shown with Fig. 2C and the SSIM criteria (Fig. 3D).

Fig. 3
figure 3

Reconstructed SR images with different protocols and the corresponding criteria compared with the SR images with a long exposure time. A SR image with a long exposure time of 3.5 ms, which was chosen as the ground truth. B SR image with a short exposure time of 0.5 ms, which was reconstructed by the Wiener deconvolution method. C SR image with a short exposure time of 0.5 ms, which was reconstructed by the Hessian reconstruction method. D SSIM criteria of SR images by the Wiener and Hessian methods compared with the ground truth (n = 16). E Magnified view of the yellow box area of A–C. Scale bars: A 2 µm; E 1 µm

Full size image

Discussion

Compared with the classic reconstruction method, the proposed protocol can estimate the parameters accurately and reconstruct SIM SR images with minimized artifacts. As experimentally demonstrated, this protocol has the dual benefits of avoiding the periodic artifacts caused by parameter errors and suppressing the artifacts caused by the underdetermined problem of the deconvolution method. Predictably, when the time resolution of the SIM microscope is further increased and the exposure time is further reduced, the proposed protocol can still suppress artifacts.

Among several parameter estimation methods, the filter method and the average method are simple improvements of the conventional parameter estimation methods but can effectively solve the problem of the bias in parameter estimation. The filter method needs to set a threshold range and may fail once the signal-to-noise ratio drops to a certain extent. The averaging method utilizes the principle that the parameters of the SIM microscope are maintained for a long period of time during the imaging process, and the N × 9 original images are averaged over time according to their phases and angles into N averaged images, which can convert the Gaussian noise to 1/√N compared with that in the original images. The iterative and non-iterative parameter estimation methods proposed by Wicker do not depend on prior information about the phase difference of the illumination vector, which is necessary when the phase difference in each direction is not certain. The image recombination transform algorithm can obtain the initial phase of fringe with high precision, even with high background and weak modulation depth.

The RL method is originally used to constrain the artifacts in the reconstructed SR image caused by the Poisson noise in the original images. However, the RL algorithm has a drawback: as the number of iterations increases, the reconstructed result does not converge. The iterative TV and Hessian algorithms converge to results with fewer artifacts but have their own advantages and disadvantages. The TV method utilizes first-order partial differentiation without decreasing the resolution of SR images but results in block structures and staircase effects in the reconstructed SR results. Thus, the TV method is suitable for biological samples that contain massive dense structures. The Hessian algorithm utilizes second-order partial differentiation to make the image more natural and is more effective in removing artifact. Moreover, the Hessian algorithm also does not depend on specific structures. Dense structures, microtubules or protein scaffolds can be applied. However, the unsuitable parameters of the Hessian algorithm result in a reduction in the image resolution. Therefore, choosing the right parameters is very important, and the recommended parameters settings are provided in a reference paper (Huang et al. 2018).

However, under the condition of a high signal-to-noise ratio, the conventional reconstruction method can well reconstruct the SR image without artifacts. Under such circumstance, using of advanced iterative deconvolution algorithms does not help to reduce artifacts further. In fact, the proposed protocol may require a more complicated procedure and more calculation time. In our opinion, it is a better choice to use the conventional reconstruction method provided by Fair-SIM under conditions of a long exposure time or high signal-to-noise ratio.

Conclusion

An ultrafast imaging rate and shorter exposure time make the signal-to-noise ratio of raw TIRF–SIM data extremely low. The noise in the raw data increases, resulting in erroneous parameter estimation and artifacts within the deconvolution results. Moreover, the raw data may be illuminated by an uneven pattern. These issues cannot be solved by a junior user of SIM microscopy. Considering this condition, we summarized the parameter estimation algorithms and deconvolution methods proposed for low signal-to-noise ratio raw data in this study. We also proposed a reconstruction protocol, combining some effective algorithms in a reconstruction procedure to minimize the artifacts of SR images. The experiment showed that the SR images reconstructed from the proposed protocol outperformed that of the commonly used method, as indicated by the evaluation criteria. We also utilized the proposed protocol to process biological images that were captured with 0.5 ms exposure times, which demonstrated fewer artifacts in reconstructed SR images than those reconstructed by the conventional Wiener method.

The given protocol enables a further reduction in the exposure time and an improvement in the time resolution. We expect it to be widely used in long-term SR imaging of live cells in the future.