1 Introduction

Fluorescence microscope is an essential tool in biological sciences thanks to its ability to visualize biomolecules of interest with great specificity and high contrast through fluorescent labeling [1,2,3]. In its classical form, however, the spatial resolution of fluorescence microscope is fundamentally limited to about 200 nm by the Abbe diffraction limit [4]. The past decades have witnessed the development of super-resolution techniques, which manage to circumvent the diffraction barrier and thus enable humans to observe the nano-scale subcellular features [5,6,7,8,9,10]. Among various super-resolution techniques, structured illumination microscopy (SIM) doubles the lateral resolution of fluorescence microscope by using spatially structured illumination to modulate high-frequency information into the passband of the microscope [8]. Notwithstanding the compromised lateral resolution with respect to other fluorescence super-resolution techniques, SIM is best suited for long-term super-resolution imaging of live cells due to its unique advantages of full-field imaging, fast imaging speed, low excitation intensity, high photon efficiency, and conventional fluorescent labels compatibility [11,12,13].

Despite all the advantages, SIM still suffers from several technical challenges that hinder its practical application and widespread adoption in the biomedical community for real-time long-term live-cell imaging. Generally, high-quality and robust SIM reconstructions rely heavily on post-processing algorithms, particularly the accurate estimation of illumination pattern parameters, such as wave vector, initial phase, and modulation depth [14, 15]. Even minor parameter errors may have a strong effect on the sensitive reconstruction results, resulting in substantial reconstruction artifacts [16,17,18,19]. Moreover, these parameters are sample- and environment-dependent, so they cannot be compensated by one-time calibration, necessitating posterior restoration from the acquired experimental data unless the experimental environment is harshly maintained stable [20, 21]. Many algorithms have been proposed to estimate illumination parameters, such as phase-of-peak (POP) [20], auto-correlation [22], non-iterative auto-correlation reconstruction (ACR) [21], and image recombination transform (IRT) [23]. These methods allow the retrieval of pattern parameters from the acquired data, thereby facilitating the artifact-free reconstruction of super-resolved images. However, their accuracy is generally limited to integer pixels, and the robustness can only be secured at a relatively low noise level. Among others, the cross-correlation approach (here we describe as COR) is probably the most popular and effective technique in regard to its robust determination of wave vectors with sub-pixel accuracy [2]. With the correlation-based optimization scheme, the wave vectors and the pattern phases can be determined with very high precision. However, its performance may still be compromised under noisy, low signal-to-noise ratio (SNR) conditions, posing a major challenge for its use for long-term live-cell imaging, where low photobleaching, phototoxicity, and light dose are a must. Moreover, the sub-pixel iterative interpolation operation is computationally lengthy and time-consuming, precluding its applications for real-time imaging of dynamic samples. Although video-rate high-speed SIM imaging has been recently demonstrated, the time-consuming parameter estimation step is not involved in the real-time workflow [24, 25]. A common practice is to calibrate the illumination parameters in advance, e.g., by using the COR algorithm, and then to use the estimated parameters for the subsequent time-lapse super-resolution reconstruction. Nevertheless, calibration at start-up requires that the imaging conditions remain invariant while continuous acquisitions are performed, which cannot follow the drift in the illumination parameters over time. Consequently, such a pre-calibration strategy may be compromised or even fails in complicated or harsh operating environments where the object or optical system cannot be maintained in a steady state. As a result, accurate, robust, efficient estimation of illumination pattern parameters in SIM is still an open quest for real-time, long-term live-cell imaging.

To this end, we present an efficient and robust SIM algorithm based on principal component analysis (PCA-SIM). Based on the observation that the ideal phasor matrix of a SIM pattern is of rank one, PCA is introduced as a “dimensionality reduction” tool for precise identification of noninteger pixel wave vector and pattern phase while rejecting components that are uncorrelated to the desired, parameter-dominating “principal component”. Experiments demonstrate that PCA-SIM achieves more accurate (0.01 pixel wave vector and 0.1\(\%\) of 2\(\pi\) relative phase) parameter estimation and superior noise immunity with more efficient non-iterative efficiency than conventional cross-correlation-based methods. The advantages of high efficiency and robustness enable flexible real-time SIM imaging of live cells based on PCA-SIM under complex experimental conditions. We also developed a MATLAB toolbox with associated datasets of our PCA-SIM algorithm, offering a ready-to-use, easy-to-operate, open-source solution for potential fast, long-term, artifact-free super-resolution live cell imaging applications.

Fig. 1
figure 1

Flow chart of structured illumination microscopy based on principal component analysis. a The raw 3-step phase-shifting SIM images and the separated spectral components. b The 1-order spectrum after being shifted \(k_{int}\). c Phases of the inverse Fourier transform of b (c1), magnified phase map from the boxed regions in c1 (c2), the ideal phases of c2 (c3) and the unwrapped version of c3 (normalized to \(-\pi\) to \(\pi\)) (c4). The sub-pixel wave vector \({{{\textbf {k}}}_{sub}}\) is reflected in the 2D phase slope of the ideal phases, but in practice, various disturbances produce serious noises. d The phases of the phasor matrix after different operations: the original phases (top first), phases after applying the masking operator (top second), phases after applying PCA (top third), and phases after least-squares fitting (top forth); and the phase distributions along the light blue line (bottom). After using the masking operator and PCA, the irrelevant components in the original phasor matrix are effectively “cleared up”. The cleaned version is close to its ideal form. e Obtained wave vector with sub-pixel accuracy (bottom), merged spectrums in one direction (middle), and merged spectrums of three directions rotating 120 degrees from each other (top). The whole process of PCA-SIM is summarized as: Step 1: obtain the Fourier spectrums of the raw SIM images; Step 2: separate the 0- and \({{\pm }}\)1-order spectrums and shift the \({{\pm }}\)1-order spectrums with integer-pixel displacement; Step 3: use a masking operator to extract the center signals for inverse Fourier transform and obtain the exponential term; Step 4: SVD and extract the principal component; Step 5: fit two principal vectors with the least square method after removing starting error points; Step 6: obtain accurate sub-pixel wave vector; Step 7: obtain the initial phase and modulation depth; Step 8: merge separated spectrums and perform super-resolution reconstruction

Full size image

2 Results

2.1 PCA-SIM principle

To break the optical diffraction limit, SIM modulates the high-frequency information of the sample, which is originally beyond the cutoff frequency of the optical transfer function (OTF), into the microscope passband by structured illumination, usually in the form of three-step phase-shifting fringe patterns. As illustrated in Fig.1a, SIM reconstruction is essentially the reorganization of diffraction-limited spectral components in the Fourier domain, consisting of 0-order spectrum [\(C_0=O({{\textbf {k}}}){{\tilde{S}}_0}({{\textbf {k}}})\)] representing the wide-field spectral information and \({{\pm }}\)1-order spectra [\(C_{{{\pm }}1}={m}O({{\textbf {k}}}){{\tilde{S}}_{ {{\pm }} 1}}({{\textbf {k}}} {{\mp }} {{{\textbf {k}}}_{ex}}){e^{ - j{\varphi _0}}}\)] for the super-resolved spectral information modulated into OTF’s support, where \({\textbf {k}}\) represents the spatial frequency coordinates, S is the desired sample function, \(\sim\) denotes the Fourier transform of the corresponding variable, O represents the system OTF, and \({{{\textbf {k}}}_{ex}}\), \({\varphi _0}\) and m are the wave vector, initial phase, and modulation depth of the sinusoidal pattern, respectively (detailed in Additional file 1: Note S1). In order to demodulate the high-frequency spectral information precisely, it is essential to reliably retrieve the illumination parameters \({{{\textbf {k}}}_{ex}}\), \({\varphi _0}\), and m, as even slight wave vector errors (within 1 pixel) and phase errors (of the order of 6\(\%\) of 2\(\pi\)) may lead to serious reconstruction artifacts [18]. We decompose \({{{\textbf {k}}}_{ex}}\) into an integer-pixel part \({{{\textbf {k}}}_{int}}\) and a sub-pixel part \({{{\textbf {k}}}_{sub}}\), where \({{{\textbf {k}}}_{int}}\) can be conveniently determined by locating the peak of the 1-order spectrum map. After a spectral shift of \({{{\textbf {k}}}_{int}}\), \(C_{{{\pm }}1}\) can be rewritten as \({m}O({{\textbf {k}}}\mathrm{{ + }}{{{\textbf {k}}}_{{\mathop \mathrm{int}} }}){\tilde{S}_1}({{\textbf {k}}} - {{{\textbf {k}}}_{sub}}){e^{j{\varphi _0}}}\) (see Fig. 1b). After inverse Fourier transform performed on \(C_{{{\pm }}1}\), the high-frequency sample information can be expressed as:

$$\begin{aligned} {\mathcal {F}^{ - 1}}[{{\tilde{S}}_1}({{\textbf {k}}} - {{{\textbf {k}}}_{sub}}){e^{j{\varphi _0}}}] = {S_1}({{\textbf {r}}}){e^{j({{{\textbf {k}}}_{sub}}{{\textbf {r}}} + {\varphi _0})}}, \end{aligned}$$
(1)

where \({\textbf {r}}\) represents the spatial coordinates. It can be seen that the sub-pixel wave vector \({{{\textbf {k}}}_{sub}}\) is exactly reflected in the 2D phase slope of the resultant phasor. Nevertheless, noise, optical aberrations, OTF-induced signal attenuation, dysregulated modulation depth, and other experimental imperfections all result in a noisy phase signal (see Fig. 1c), which prohibits reliable determination of the sub-pixel wave vector by using traditional methods, including COR (detailed in Additional file 1: Note S2). Therefore, eliminating these irrelevant disturbances is the key to the success of robust parameter estimation for SIM under low SNR conditions.

To devise a solution to such a challenging problem, we first consider an ideal case that is free from all above-mentioned interferences, and the phasor of an ideal sinusoidal pattern (free from sample modulation) can be denoted as \({e^{j({{{\textbf {k}}}_{sub}}{{\textbf {r}}} + {\varphi _0})}}\). A close inspection of the ideal pattern phasor function reveals that it is essentially a rank-one matrix, which can be represented as the product of two vectors:

$$\begin{aligned} exp[j({{{\textbf {k}}}_{sub}}{{\textbf {r}}} + {\varphi _0})]= {{{\textbf {s}}}_x}{{\textbf {s}}}_y^H, \end{aligned}$$
(2)

where \({{{\textbf {s}}}_x} = exp[j({{{\textbf {k}}}_{x,sub}}{{{\textbf {r}}}_x} + {\varphi _{x,0}})]\), \({{{\textbf {s}}}_y} = exp[-j({{{\textbf {k}}}_{y,sub}}\) \({{{\textbf {r}}}_y} + {\varphi _{y,0}})]\), the subscripts x and y indicate the components along the horizontal and vertical directions, and \({\{ \cdot \} ^H}\) represents the complex-conjugate transpose. Such a decomposition inspires us that the ideal pattern phasor matrix \(exp[j({{{\textbf {k}}}_{sub}}{{\textbf {r}}} + {\varphi _0})]\) should have only one principal component, which describes the single best subspace of the data in the least-squares sense. However, under real conditions, experimental imperfections and other disturbances will inevitably produce noisy measurements, resulting in high dimensionality of the pattern phasor matrix. Then the extraction of \({{{\textbf {k}}}_{sub}}\) and \({\varphi _{x,0}}\) from the noisy measurements is recast as finding the first principal component of the high-dimensional pattern phasor matrix, which is precisely the task accomplished by principal component analysis (PCA) [26]. PCA is the process of reducing data dimensionality by geometrically projecting them onto lower subspace called principal components, with the goal of finding the best summary of the data using a limited number of principal components (here only the first principal component is kept and the rest is ignored). In PCA, the principal components are eigenvectors of the data’s covariance matrix. Thus, the principal components are often computed by singular value decomposition (SVD) of the data matrix. The 2D slope, i.e., the sub-pixel wave vector (\({{{\textbf {k}}}_{x,sub}}\), \({{{\textbf {k}}}_{y,sub}}\)) in \({{{\textbf {s}}}_x}\) and \({{{\textbf {s}}}_y}\), can then be independently determined from 1D unwrapped phase distribution of the left and right dominant singular vectors based on linear regression. Without reference to prior knowledge, the unwanted noise and other disturbances in the raw pattern phasor matrix are effectively “cleaned up” by PCA after dimensionality reduction, as illustrated in Fig. 1d. More details about the principle and implementation of the PCA-SIM algorithm are summarized as Algorithm 1 in Additional file 1: Note S3.

Though PCA-SIM improves the accuracy and robustness of parameter estimation without using interpolation, PCA itself can be a time-consuming operation for relatively large matrix dimensions. In addition, real-time, long-term live-cell imaging applications require low photobleaching, phototoxicity, and therefore very limited light dose. Under such low SNR conditions, it remains a major challenge to accurately extract the eigenvectors and eigenvalue of the first principal component from highly noised data. To further accelerate PCA-SIM and enhance its noise robustness, we introduce an additional frequency-domain masking operator on the noisy 1-order spectrum before applying PCA. The masking operator can be further divided into the inner signal window (for retaining the main signal) and the external padding region (for denoising). In Additional file 1: Note S4, we derive that the Fourier spectrum of an ideal pattern phasor matrix \(exp[j({{{\textbf {k}}}_{sub}}{{\textbf {r}}} + {\varphi _0})]\) with a sub-pixel wave vector is a downsampled 2D Dirichlet function with most of its energy is concentrated in limited support around the integer part of the wave vector (see Additional file 1: Fig. S1). Therefore, a frequency domain mask is applied to the noisy 1-order spectrum to retain the high-energy spectral peak region while zeroing out the low SNR data region outside this window, as illustrated in Additional file 1: Fig. S2. Because the limited mask almost encompasses the dominant energy of the downsampled 2D Dirichlet function (concentrated within small support) while the noise energy (that spread evenly over the entire 1-order spectrum) is significantly suppressed, significant improvement in accuracy and robustness of parameter estimation can be expected, especially under low SNR conditions (see Additional file 1: Fig. S3–S5 for more details). In addition, the masking operator significantly reduces the amount of data involved in the PCA computation, and thus the computational efficiency of the whole algorithm can be greatly improved. More details about the implementation of the PCA-SIM algorithm with the application of frequency domain masking operator are summarized as Algorithm 2 in Additional file 1: Note S5.

After \({{{\textbf {k}}}_{sub}}\) is determined, the initial phase \({\varphi _0}\) can be obtained from the phase angle of \(exp[j({{{\textbf {k}}}_{sub}}{{\textbf {r}}} + {\varphi _0})]\) at \({{\textbf {r}}} = 0\). The modulation depth m is then calculated by complex linear regression of \(C_0\) against the sub-pixel shifted version of \(C_{{{\pm }}1}\). Then, the 0- and \({{\pm }}\)1-order spectrums can be accurately separated and merged in the frequency domain, as illustrated in Fig. 1e. To achieve isotropic super-resolution, a similar operation is performed on two other sets of SIM images with fringe orientations rotating by \({{\pm }}\) 120 degrees (see Fig. 1e). Finally, an image deconvolution algorithm (e.g., Wiener reconstruction [8], total variance (TV) [27], or High-Fidelity (HiFi) SIM [15]) is further applied on the synthetic spectrum to compensate the effects of the OTF and spectral overlapping regions, resulting in a super-resolved image of the sample. The complete flow diagram of the PCA-SIM algorithm is summarized in Fig. 1.

Fig. 2
figure 2

Comparative simulations of different methods for parameter estimation and super-resolution reconstruction. a The wave vector errors of PCA-SIM and COR under different noise environments. b The initial phase errors of different methods (PCA-SIM, COR, POP, ACR and IRT) under different noise environments. c The wide-field images of the samples under different noise conditions and the super-resolution image reconstructed by PCA-SIM. Images in the middle circular area are the wide-field images when the noise power is 25dBW and 45dBW respectively, and the image in the surrounding rectangular area is the super-resolution image acquired by PCA-SIM when the noise is 25dBW. The magnified label image corresponding to the blue boxed region is shown in the top left. d Magnified wide-field images (left) and super-resolution images from the boxed regions in c obtained by different methods when the noise is 25dBW (‘None’ means no parameter estimation). To quantify the reconstruction quality, we calculate the structural similarity index measure (SSIM) between the reconstruction results and Label, which are 0.8377, 0.8383, 0.8468, 0.9736 and 0.9739 for POP, ACR, IRT, COR, and PCA, respectively. e Magnified wide-field images (left) and super-resolution images from the boxed regions in c obtained by different methods when the noise is 35dBW. The SSIM are 0.7924, 0.7919, 0.8052, 0.9152 and 0.9661 for POP, ACR, IRT, COR, and PCA, respectively. f Magnified wide-field images (left) and super-resolution images from the boxed regions in c obtained by different methods when the noise is 45dBW. The SSIM are 0.6872, 0.6785, 0.6850, 0.7177 and 0.9451 for POP, ACR, IRT, COR, and PCA, respectively. g Intensity profiles along the light blue line in d-f (normalized to maximum). Simulations were repeated ten times independently with similar results. Scale bars: 1 \(\mu\)m (c)-(f)

Full size image
Fig. 3
figure 3

Comparative experiments on super-resolution results of a COS-7 cell sample with DAPI-labeled nucleus, Alexa Fluor\(^{TM}\) 568-labeled actin and MitoTracker\(^{TM}\) Green FM-labeled mitochondria. a Comparison of the wide-field image and the super-resolution image obtained by PCA-SIM. The wide-field image is displayed in the lower left triangle, which was obtained by averaging 9 raw SIM images (with the resolution of \(1024 \times 1024\)) captured through a 100\(\times\) objective (UPlanSApo 100\(\times\)/1.40 Oil, Olympus, Japan). The super-resolution image reconstructed by PCA-SIM is shown on the upper right. For easy distinguishing, we show the mitochondria, actin and nucleus of the COS-7 cell in red, green and blue, respectively, where the mitochondria are excited at 488 nm wavelength, the nucleus at 405 nm wavelength and the actin at 561 nm wavelength. b Magnified wide-field image (top left) and super-resolution images of actin from the yellow boxed regions in a obtained by different methods (POP, ACR, IRT, COR and PCA-SIM). c Magnified wide-field image (top left) and super-resolution images of mitochondria from the blue boxed regions in a obtained by different methods (POP, ACR, IRT, COR, and PCA-SIM). d Intensity profiles along the light blue line in b and c (normalized to maximum). Experiments were repeated ten times independently with similar results. The time-consuming of estimating parameters by COR and PCA is 16.3227 \({{\pm }}\) 0.0479 seconds and 0.8332 \({{\pm }}\) 0.0003 seconds, respectively. Colored arrows point to regions where reconstruction differences are distinct. Scale bars: 5 \(\mu\)m (a); 200 nm (b); 200 nm (d)

Full size image

2.2 Simulations to demonstrate the superior comprehensive performance of PCA-SIM in terms of accuracy, efficiency, and noise immunity

A set of comparative simulations were carried out to verify the effectiveness of PCA-SIM. We used a high-resolution (\(1024 \times 1024\) pixel) image of bovine pulmonary artery endothelial (BPAE) cells with rich details, sharp edges, and smooth background as the ground truth. The high-resolution image was multiplied by ideal structured illumination patterns with frequencies matching the incoherent diffraction limit of a 40\(\times\) 0.6 NA objective lens. Then, its Fourier spectrum was attenuated with a corresponding OTF to simulate the ideal SIM images acquired by a 40\(\times\) 0.6 NA microscope (note that the main results obtained via these simulations do not significantly depend on the choice of imaging parameters). Finally, Gaussian noises with different powers were artificially added to generate the raw SIM images under different SNR conditions. Figure 2a shows the performances of PCA-SIM and COR for estimating wave vectors at different noise levels. It can be seen that PCA-SIM can obtain a more precise wave vector with an accuracy better than 0.01 pixel at low noise levels. When the noise level increases to 40 dBW, COR yields serious errors larger than 1 pixel, while the errors of PCA-SIM can still be controlled within 0.2 pixels. The initial phase errors of different methods at different noise levels are illustrated in Fig. 2b, from which we can see that the accuracy of PCA-SIM can stably achieve 0.1\(\%\) of 2\(\pi\) under typical noise conditions, thanks to its reliable wave vector estimation. In contrast, the POP, ACR, and IRT methods cannot provide high-accuracy phase estimation, because their wave vector estimation is limited to integer pixels only. Figure 2c shows the wide-field images of the sample under different noise conditions and the super-resolution reconstruction obtained by PCA-SIM. Figures 2d, f illustrate the super-resolution reconstruction comparisons of different methods at three low-to-high noise levels (25 dBW, 35 dBW, and 45 dBW). When the noise is less than 40 dBW, both PCA-SIM and COR yield good results, while the results of other methods suffer certain degrees of distortions. When the noise is greater than 40 dBW, the results obtained by COR deteriorate while PCA-SIM remains visually close to the ground truth. Note that the SIM reconstruction results shown in Figs. 2d-2f are obtained by using parameters estimated from different noise data but noise-free raw SIM images for better visual comparison. It should be also noted that the time cost of PCA-SIM for parameter estimation is only 0.8119 \({{\pm }}\) 0.0011 s, processed with MATLAB R2016a using the open source code in a Dell XPS 8930 computer (Intel(R) Core(TM) i7-9700 CPU, NVIDIA GeForce GTX 1660 Ti), while that of COR is 16.2810 \({{\pm }}\) 0.0355 s. In addition, we also provide the simulation comparisons of the standard simple structure, which can more intuitively demonstrate the advancement of PCA-SIM (Additional file 1: Fig. S6). These results demonstrate that parameter estimation has a considerable impact on the fidelity of SIM reconstruction, and PCA-SIM has the best performance in terms of accuracy, efficiency, and noise immunity, which in turn provides the highest quality image reconstructions than the state-of-the-art technologies.

Fig. 4
figure 4

Comparative experiments on super-resolution results of a HeLa cell sample with DAPI-labeled nucleus and FITC-labeled \(\alpha\)-SMA at different SNRs. a Comparison of the wide-field image and the super-resolution image obtained by PCA-SIM. The wide-field image is displayed in the upper right triangle, which was obtained by averaging 9 raw SIM images (with the resolution of \(1024 \times 1024\)) captured through a 60\(\times\) objective (\({\textbf {UPlanXApo 60}}\times /{\textbf {1.42 Oil}}\), Olympus, Japan). The super-resolution image reconstructed by PCA-SIM is shown on the bottom left. We show the \(\alpha\)-SMA and nucleus in green and blue, respectively, where the \(\alpha\)-SMA are excited at 488 nm wavelength, and the nucleus at 405 nm wavelength. b Magnified wide-field image (top) and super-resolution images from the boxed regions in a obtained by COR and PCA-SIM. c, d Magnified wide-field images (top) and super-resolution images from the yellow boxed regions in a obtained by COR and PCA-SIM under different noise environments. e The wide-field images under different noise environments (since wide-field images are obtained by multi-frame averaging, the quality of raw SIM images with noises is actually worse, and when the noise power reaches 70 dBW, the sample signals are almost completely drowned). f Structural similarity indexes of the super-resolution results of COR and PCA-SIM under different SNRs with respect to b, respectively. Experiments were repeated thirty times independently with similar results. Colored arrows point to regions where reconstruction differences are distinct. Scale bars: 10 \(\mu\)m (a), (e) ; 1 \(\mu\)m (b)-(d)

Full size image

2.3 Comparative experiments to demonstrate the superior super-resolution reconstruction capability of PCA-SIM

We constructed a tri-color laser-interference SIM system (Additional file 1: Fig. S7) based on an off-the-shelf inverted fluorescence microscope (IX73, Olympus, Japan) to compare the performance of PCA-SIM with COR, POP, ACR, and IRT through practical experiments. A CV-1 in Origin Simian-7 (COS-7) cell sample (with DAPI-labeled nucleus, Alexa Fluor\(^{TM}\) 568-labeled actin and MitoTracker\(^{TM}\) Green FM-labeled mitochondria) was fixed, and the raw SIM images were acquired by our SIM system. As shown in Fig. 3, though the image resolution is greatly improved in the results of POP, ACR, and IRT, the fine details of the mitochondria and actin filament are still blurred. In contrast, these fine structures are resolved with higher contrast and fidelity by COR and PCA-SIM, benefiting from more accurate estimates of illumination parameters. Note that when the difference of wave vector estimate is within 0.1 pixel, the difference in the resultant reconstruction can hardly be distinguished visually. Further analysis of the fluorescence intensity profile distributions in the super-resolution results reconstructed by different methods reveals a slight resolution improvement of PCA-SIM over COR (Fig. 3d). The other comparative results of COS-7 cells using different objective configurations also demonstrate that PCA-SIM yields super-resolution images with better quality (Additional file 1: Fig. S8). In addition, we applied PCA-SIM to different samples collected by different SIM microscopes, e.g., BPAE cells collected by our experimental setup (Additional file 1: Fig. S9), the autofluorescent ascaris sample collected by a commercial SIM system (N-SIM, Nikon, Japan) (Additional file 1: Fig. S10), microtubules in COS-7 cells collected by another commercial SIM system (GE DeltaVision OMX, GE, USA, data available in the literature [15] (Additional file 1: Fig. S11), and f-actin in COS-7 cells acquired by a 1.7 NA total internal reflection fluorescence (TIRF) objective (data available in the literature [11], Additional file 1: Fig. S12). In all these experimental datasets, PCA-SIM outperforms other approaches and achieves high-quality super-resolution reconstruction with higher efficiency.

Fig. 5
figure 5

Real-time super-resolution reconstructions of mitochondria of live COS-7 cells labeled by MitoTracker\(^{TM}\) Green FM at different time points. a, d Comparison of wide-field images and super-resolution images obtained by PCA-SIM. Wide-field images are displayed in the bottom left triangle, which were obtained by averaging 9 raw SIM images (with the resolution of \(512 \times 512\)) captured through a 60\(\times\) objective (UPlanXApo 60\(\times\)/1.42 Oil, Olympus, Japan). Super-resolution images reconstructed by PCA-SIM are shown on the upper right. b, e Magnified super-resolution images from the blue boxed regions in a and d at different time points. c, f Magnified super-resolution images from the white boxed regions in a and d at different time points. Colored arrows point to regions where dynamics are distinct. Scale bars: 5 \(\mu\)m (a), (d); 1 \(\mu\)m (b) left, (c), (e) and (f); 0.5 \(\mu\)m (b) right

Full size image

2.4 Experiments to demonstrate the noise robustness of PCA-SIM

The potential of SIM for live-cell imaging is often difficult to be fulfilled owing to photobleaching and phototoxicity that can substantially perturb the desired physiology. Reducing the excitation power can effectively mitigate the photodamage, but the resulting low SNR poses a significant challenge to the post-processing algorithm. Here, we further verify that PCA-SIM is robust to low SNRs through experiments on fixed HeLa cells with DAPI-labeled nucleus and FITC-labeled \(\alpha\)-SMA (smooth muscle actins). Considering that photobleaching would make it difficult to image the sample repeatedly hundreds of times under dense SNR conditions, we artificially added simulated Gaussian noises to the practically captured raw SIM images to produce different SNR situations. As SNR decreases, the structural distortion and contrast imbalance in the super-resolution images reconstructed with COR increase significantly (Fig. 4a–e). In contrast, these artifacts are effectively suppressed by PCA-SIM, yielding high-quality results almost visually identical to those at high SNR. Note that here we also used the noise-added low-SNR images for parameter estimation and the raw captured images for image reconstruction to better visualize the impact of the parameter estimation. To further quantify the quality of super-resolution images obtained by COR and PCA-SIM, we calculated the SSIM value of the reconstruction results at different SNRs with respect to those at high SNR (without artificial noise). In constructing SIM images with added noise from 60 dBW to 80 dBW, the super-resolution images reconstructed with COR suffer a 92\(\%\) reduction in SSIM value (medians decreased from 0.99 to 0.07), while PCA-SIM only has a mild reduction (0.97 to 0.6) (Fig. 4f). In addition to Gaussian noise, PCA-SIM also demonstrates strong noise resistance under Poisson noise conditions, and the general trend of the SSIM distributions does not differ significantly from the cases of Gaussian noise (Additional file 1: Fig. S13). In Additional file 1: Fig. S14, we generate the noisy dataset on a BPAE sample under different SNR conditions by alternatively reducing the excitation powers. The corresponding experimental results coincide with those of simulations with artificially added noise, indicating that PCA-SIM provides more robust reconstruction quality at low SNR. All these experimental results demonstrate the potential of PCA-SIM for application to long-term live-cell imaging in low-excitation conditions.

2.5 Real-time live-cell SIM super-resolution imaging with illumination parameters correction

The high accuracy, robustness, and efficiency of PCA-SIM open the possibility of on-the-fly super-resolution reconstruction with illumination parameters compensation. Such an instant parameter updating strategy has the potential to significantly improve the live-cell imaging performance of SIM under imperfect imaging conditions with external disturbances. To validate this, we achieved real-time PCA-SIM acquisition, reconstruction, and display based on graphics processing unit (GPU) acceleration, and used it to measure the dynamics of mitochondria of live COS-7 cells labeled by MitoTracker\(^{TM}\) Green FM. Recent studies have indicated that mitochondrial dynamic tubulation (MDT) drives the formation of mitochondrial networks, which are essential for maintaining mitochondrial function (mitochondrial DNA integrity, apoptosis, etc.) [28, 29]. However, MDT tubules are approximately 100 nm in diameter and are highly dynamic, posing a challenge to the spatial and temporal resolution of the imaging. In our results, the dynamic events of mitochondrial tubules are well reconstructed by PCA-SIM (Fig. 5). We observed that the MDT tubule, different from the regular thicker and more static mitochondria, extended rapidly from the mitochondria (Fig. 5a, b). When fusing with another mitochondrion and forming a membrane bridge between them, the MDT tubule thickened and became the mitochondrial network, during which the transfer of material occurred (white dashed circle in Fig. 5b). More complex dynamic tubule events are recorded in Fig. 5c, where an MDT tubule grew a new one after fusing with the left mitochondrion (4.32 s), and the new tubule then fused with another mitochondrion (9.09 s). Figures 5d–f also illustrate interesting mitochondrial dynamics, where one mitochondrion was significantly deformed due to being dragged by an MDT tubule (Fig. 5e), and an extended tubule turned nearly 90 degrees in search of a mitochondrion with which to fuse (Fig. 5f). Additional file 1: Movie S1 provides the complete dynamic SIM super-resolution reconstruction results as well as the raw SIM images for comparison. We realized a reconstruction frame rate of about 11.1 frames per second with 30 ms exposure time based on the constructed SIM system. Despite frequent switching of the region of interest and adjustment of the focus, with some consequent environmental perturbations, PCA-SIM always maintained high imaging quality thanks to its ability to extract accurate experimental parameters from the raw SIM images with low SNRs per reconstruction. In addition, dynamic imaging scenes and results of a fixed BPAE cell sample with ambient light interference and artificial perturbations also demonstrate that PCA-SIM can provide immediate, high-quality super-resolution reconstruction in a complicated environment (Additional file 1: Fig. S15, and Additional file 3: Movie S2). Compared to the COR-based parameter calibration in advance, which caused subsequent reconstruction failures when the experimental environment varies, such as operators interacting with the experiment based on the results, PCA-SIM achieved adaptive parameter estimation, thus always obtaining stable and high-quality super-resolution images (Additional file 1: Fig. S16).

3 Discussion and conclusion

In summary, we have developed an efficient and robust SIM parameter estimation and image reconstruction approach based on PCA (PCA-SIM). By extracting the first principal component of the pattern phasor matrix, the noise and other interference components can be effectively removed, leading to precise retrieval of noninteger pixel wave vector and pattern phase in a simple and efficient manner. Compared with state-of-the-art SIM algorithms, including the established COR, PCA-SIM has demonstrated superior performance in terms of estimation accuracy, image quality, efficiency and noise robustness. Based on PCA-SIM, we implemented real-time SIM reconstruction in complex experimental scenarios, which allowed the operator to give feedback on the experiment instantly based on the displayed image, facilitating the utilization of the SIM microscope. In addition, we provide an associated open-source and easy-to-use MATLAB toolbox to help users process, analyze, and visualize SIM data based on PCA-SIM (Additional file 4). Simulations and experimental results suggest that PCA-SIM is a promising method for real-time, long-term, high-quality live-cell imaging under low-SNR, weak-excitation conditions.

It should be emphasized that the current work is mainly focused on parameter estimation algorithm in SIM. By further increasing the hardware speed, it is expected to achieve more rapid SIM imaging. Under extremely low SNR conditions, PCA-SIM can be combined with regularization techniques, e.g., Hessian-SIM [18], to gain additional capabilities of image denoising by multi-frame averaging under a priori of spatio-temporal continuity. Finally, while only 2D SIM parameter estimation is addressed here, PCA-SIM can be easily generalized to 3D SIM by extending PCA and associated operations from 2D to 3D. Since the data volume of 3D SIM (intensity stack) is much larger than that of 2D SIM, we believe that the improvement in accuracy and speed of parameter estimation over the traditional COR method [2] brought by the dimensionality reduction feature of PCA should be more significant. We leave this generalization of 2D PCA-SIM to 3D as future work.

4 Methods

4.1 SIM setup

Our tri-color SIM setup is constructed based on an off-the-shelf inverted fluorescence microscope (IX73, Olympus, Japan), as shown in Additional file 1: Fig. S7. Three laser beams (Laser 1: OBIS LX405, Coherent, USA; Laser 2: OBIS LX561, Coherent, USA; and Laser 3: Sapphire 488LP-200, Coherent, USA) are coupled by a plane mirror (M1) and two dichromatic mirrors (DM1: ZT561dcrb, Chroma, USA; and DM2: ZT488dcrb, Chroma, USA), and then filtered, expanded and collimated by a spatial filter and a collimating lens (L1: LSB08-A 150 mm, Thorlabs, USA). After being modulated by a half-wave plate (H1: GCL-0604, Daheng Optics, China) into p(parallel)-polarized light in order to reduce the subsequent light intensity loss, the laser beam passes through a polarization beam splitter (PBS: PBS251, Thorlabs, USA) and is projected onto a ferroelectric liquid crystal spatial light modulator (SLM: QXGA-3DM, Fourth Dimension Displays, UK) displaying dense gratings. The spatial light modulator is used to generate diffraction of light, which is then modulated to s(senkrecht)-polarized light by another half-wave plate (H2: GCL-0604, Daheng Optics, China) and reflected again through the polarization beam splitter. An achromatic lens (L2: LSB08-A 250 mm, Thorlabs, USA) converges diffracted beams to its focal plane, or spectral plane, with a mask designed in advance for blocking the 0-order beam and permitting the passage of the \({{{\pm }}1}\)-order diffracted light only. A polarization rotator (PR) is usually introduced to maximize the modulation contrast of the illumination pattern. The \({{{\pm }}1}\)-order diffracted light then passes through a lens pair (L3: LSB08-A 200 mm, Thorlabs, USA; L4: LSB08-A 175 mm, Thorlabs, USA), focuses on the back focal plane of the objective lens (OBS: UPlanSApo 100\(\times\)/1.40 Oil, Olympus, Japan; UPlanXApo 60\(\times\)/1.42 Oil, Olympus, Japan), and interferes in the sample plane. The emission fluorescence is collected by the same objective, propagated through the fluorescent interference filter block (FM: U-FVN, Olympus, Japan; U-FBW, Olympus, Japan; U-FYW, Olympus, Japan) and a tube lens (TL), and finally captured by an sCMOS camera (PCO Edge 5.5, PCO, Germany) with 60\(\%\) quantum efficiency to obtain fluorescent images, whose exposure is triggered by the spatial light modulator.

4.2 Image processing

The raw SIM images were first slightly edge-attenuated as a precaution against edge-related artifacts [2]. Before parameter estimation, the Richardson-Lucy deconvolution was applied to remove partial Poisson noise and out-of-focus background [30, 31]. For parameter estimation, PCA-SIM removed the interference signals unrelated to the illumination parameters from the phasor matrix (\(exp[j({{{\textbf {k}}}_{sub}}{{\textbf {r}}} + {\varphi _0})]\)) by SVD and using a masking operator, thus achieving efficient and accurate experimental parameter extraction. More details about the principle of PCA-SIM are summarized in Additional file 1: Notes S3–S5. For other comparison methods, COR followed the procedure in fairSIM [19], and POP, ACR and IRT were performed with reference to the literature [20, 21, 23]. For image reconstruction, we used the reconstruction algorithm of the literature [15], which is an improved Wiener reconstruction algorithm by point-spread-function engineering. All image processing of static results was implemented using customized codes in MATLAB R2016a (except for the image labeled ‘N-SIM’ in Additional file 1: Fig. S10, which was reconstructed with NIS-Elements). All image processing of dynamic results was implemented using developed SIM software in Visual Studio 2017.

4.3 GPU-accelerated SIM reconstruction

Since the SIM reconstruction algorithm contains numerous pixel-wise matrix operations and (inverse) Fourier transforms, it is well-suited and easily accelerated by GPU. We take advantage of GPU to implement the reconstruction through customized kernel functions based on CUDA, which is a parallel computing architecture developed by NVIDIA. For the convenience of the hardware control, we redevelop them in our own software, where exposure adjustment and image acquisition are realized through the Software Development Kit (SDK) of the sCMOS camera, and the communication between the lasers, SLM and the computer is established through the serial interface and the corresponding commands. All software programs are integrated into Qt framework (a cross-platform software for creating graphical user interfaces) based on C++ language. The software contains one main thread and several sub-threads that run in parallel with each other. The main thread is used for the creation and visualization of the graphical user interface (GUI), the parameter setting of the hardware and the communication between the threads. In the camera thread, the raw SIM images captured by the sCMOS camera are grabbed into a ring buffer, and simultaneously the corresponding image sequence numbers are added. In the reconstruction thread, each 9 raw SIM images in the ring buffer required for super-resolution reconstruction, which can be determined by the image sequence, are copied from the host memory (CPU) to the device memory (GPU), and then processed by a series of custom kernel functions to obtain the super-resolution image. In order to improve the temporal smoothness of the display, the sliding window strategy is adopted. Except for the first reconstruction which requires 9 images in the image buffer, subsequent reconstructions can be performed with only three images (raw images in a certain direction) updated. In the display thread, the reconstructed super-resolution image is copied back to CPU and transferred to the main thread along with the first of 9 raw images for display on GUI.

4.4 Sample preparation

The fixed COS-7 cell sample was obtained from Smart Computational Imaging Research Institute (SCIRI) of Nanjing University of Science and Technology, and cultured in Dulbecco’s modified Eagle’s medium (DMEM) on 0.17 mm coverslips, mixed with 10\(\%\) fetal bovine serum (FBS) and 1\(\%\) 100 mM sodium pyruvate solution at 37\(^{\circ }\)C and 5\(\%\) CO\(_2\). MitoTracker\(^{TM}\) Green FM (Thermo Fisher) was used to label mitochondria in DMEM (Gibco) at 37\(^{\circ }\)C for 30 min. After fixation, permeabilization and washing, F-actin is labeled by Alexa FluorTM 568 (Thermo Fisher) at 37\(^{\circ }\)C for 45 min and nucleus was labeled by DAPI (Thermo Fisher) at 37\(^{\circ }\)C for 30 min. Then the sample is sealed with glycerol after washed by Phosphate Buffered Saline (PBS) and pure water.

The fixed BPAE cell sample was obtained from Cell Applications, Inc. and cultured in Bovine Endothelial Cell Growth Medium in a 37\(^{\circ }\)C, 5\(\%\) CO2 humidified incubator. MitoTracker\(^{TM}\) Red CMXRos (Thermo Fisher) was used to stain mitochondria at 37\(^{\circ }\)C for 30 min. After fixation, permeabilization and washing, F-actin was stained with Alexa Fluor\(^{TM}\) 488 (Thermo Fisher) phalloidin at 37\(^{\circ }\)C for 45 min, and the nucleus was counterstained with blue fluorescent DNA staining DAPI (Thermo Fisher) at 37\(^{\circ }\)C for 30 min.

The HeLa cells were obtained from Smart Computational Imaging Research Institute (SCIRI) of Nanjing University of Science and Technology, and cultured in Minimum Essential Medium (MEM), mixed with 10\(\%\) FBS and 1\(\%\) Penicillin/Streptomycin (P/S) at 37\(^{\circ }\)C and 5\(\%\) CO2. The slice was dropped into anti-alpha smooth muscle Actin antibody (primary antibody, Abcam) added with PBS and incubated overnight in a wet box at 4\(^{\circ }\)C, then dropped into the Goat Anti-Mouse IgG H &L (secondary antibody, Abcam) and incubated in a dark greenhouse for 50 minutes. After being placed in PBS (pH 7.4), shaking and washing on the decolorization shaking table for 3 times, the nucleus was labeled by DAPI in a dark greenhouse for 10 min.

The live COS-7 cells were incubated in H-DMEM containing 10\(\%\) FBS and 1\(\%\) penicillin-streptomycin under a humidified environment of 5\(\%\) CO2 at 37\(^{\circ }\)C. COS-7 cells were seeded at a density of 5 \(\times\) \(10^8\) cells in the confocal dishes with three parallel samples, which were cultured overnight. MitoTracker\(^{TM}\) Green FM (Beyotime, China) were diluted to 200 nM in H-DMEM, which should be pre-warmed at 37\(^{\circ }\)C. They were washed by 0.5 mL PBS for 3 min. The prepared solution was incubated for 10 min at a dose of 0.5 mL/well under dark conditions. To remove interference from free dyes, all samples were washed by PBS for 3 times.

4.5 Statistical analysis

Except for Fig. 4, Additional file 1: Figs. S14–S16, all the figures showed the representative data from 10-30 representative experiments. The experimental parameter errors and the structural similarity index of the reconstruction results in Figs. 2, 4, Additional file 1: Figs. S3, S4, S6 and S13 were presented as box plots (center line, average; limits, 75\(\%\) and 25\(\%\); whiskers, maximum and minimum) in graphs. The intensity profiles in Figs. 2, 3, and Additional file 1: Fig. S6 are interpolated by linear interpolation in MATLAB. The running time of parameter estimation of COR and PCA-SIM was calculated by the stopwatch timer function in MATLAB, and was expressed as mean \({{\pm }}\) SEM (standard error of the mean). Note that these times were evaluated in the case of processing data through MATLAB R2016a in a Dell XPS 8930 computer (Intel(R) Core(TM) i7-9700 CPU, NVIDIA GeForce GTX 1660 Ti) using open source code.