Introduction

The advent of digital fluorescence microscopy and the development of fluorescent proteins allows for unprecedented observation of molecular behavior during complex cellular processes. Because visualization of fluorescent probes is possible in live cells, not only can the cellular location of proteins be studied, but in vivo molecular and cellular protein dynamics can be observed. A number of sophisticated techniques, including fluorescence recovery after photobleaching, fluorescence resonance energy transfer, and fluorescence correlation spectroscopy have been developed to further utilize fluorescence technology.

Although fluorescently labeled proteins can localize to cellular components with a high degree of specificity, the spatial resolution of fluorescence microscopy is limited by the diffraction of light from a fluorophore and by the brightness of the fluorophore against background fluorescence. The spatial resolution of fluorescence microscopy is inherently limited by the wavelength of visible light, with a maximum achievable lateral resolution of ~200 nm and longitudinal resolution of ~500 nm (the “Abbe limit”) for commonly used oil-immersion objective lenses having a numerical aperture of 1.4.15,28 In addition, conventional fluorescence microscope images contain information from out-of-focus planes both above and below the focal plane, which further complicates the analysis.1 While confocal fluorescence microscopy can reduce this contribution from out-of-focus fluorescence, it suffers from increased noise as well as from object distortion in the plane of focus.27 Background noise reduces the maximum achievable resolution of spatially separated fluorophores, especially in the case of dim fluorescent signals. Thus, although fluorescent markers can precisely localize cellular proteins and structures, difficulties in resolving individual same-color fluorophores through standard digital fluorescence microscopy hinders accurate quantitative analysis of feature locations. This is particularly true for live-cell imaging of proteins fused to green fluorescent protein (GFP). New optical methods are being developed that break through the Abbe limit5,6,12,13 for 3D imaging, but practical considerations so far limit resolution to ~100 nm.

The ability to validate theoretical spatial models for the location of cellular components at resolutions beyond the Abbe limit would increase the utility of fluorescence microscopy, allowing for improved quantitative analysis and for modeling of dynamic cellular processes. Theoretical models can be developed via qualitative analysis of fluorescence images, complementary experimental methods such as electron microscopy, or computer modeling. In previous work, we have applied the model-convolution method to validate a variety of theoretical models,3,4,10,11,25 and found that the model-convolution approach provides a seamless and objective method to directly compare theoretical models to experimental results. In addition, we have found that simulated images generated through the model-convolution method can be used to evaluate the reliability of experimental measurement methods for a particular application, such as in evaluating the accuracy of Gaussian fitting to track beads or evaluate filament curvature distributions.2,4

Experimental Deconvolution in Quantitative Fluorescence Microscopy

A typical solution used to reduce blur and out-of-focus fluorescence and thereby improve spatial resolution in conventional fluorescence microscopy is image deconvolution.21 The spreading of light by diffraction through a microscope lens relative to the focal plane of the point light source is termed the “point spread function” (PSF), and can be either experimentally measured or theoretically calculated for a given microscope. Computational deconvolution methods use the experimentally measured or theoretically predicted PSF to deblur fluorescence images. Deconvolution methods work by estimating and then removing the contribution of light both from out-of-focus fluorescence and from in-plane spreading of light due to diffraction. Although image deconvolution can be successful in improving image detail and contrast, as well as in reducing background haze,21 it is unclear whether this method is always appropriate for quantitative spatial characterization in images containing same-color multiple fluorescent protein copies (>2 copies). Specifically, the presence of noise in the data makes it difficult to exactly reconstruct the underlying fluorophore distribution, so that deconvolution can only produce the distribution in a statistical sense, often via least-squares minimization.21 In addition, the interpretation of deconvolved images can be problematic, as the computational deconvolution operation is performed without regard to known physical or molecular information about the system being studied.

We illustrate a potential pitfall of deconvolution with an example that involves detection of the positions of kinetochores between spindle poles within the budding yeast mitotic spindle. A simulated distribution was made that included 32 green fluorophores (representing 32 kinetochores) and 2 red fluorophores (representing two spindle pole bodies), located along a ~1500 nm spindle axis. This is the fluorophore distribution predicted by electron microscope reconstructions of a typical budding yeast mitotic spindle22,31 (Figs. 1a and 1b). A simulated fluorescence image of the mitotic spindle was generated by convolving this fluorophore distribution with the wide-field microscope PSF and then by adding a level of background noise to the image that is typical of budding yeast cells expressing proteins fused to GFP (Fig. 1c). This simulated image was deconvolved using standard constrained iterative deconvolution software (DeltaVision, Fig. 1d). Resolution of the two widely spaced red fluorophores at the spindle poles is clearly successful in the deconvolved image. Thus, experimental deconvolution is useful in this instance for contributing to accurate measurement of spindle pole locations (through Gaussian fitting or similar), in assessing the relative brightness of the poles, and in confirming that there are indeed two fluorescent objects present in the image.

Figure 1
figure 1

Image deconvolution cannot always resolve individual fluorophore locations. (a) A typical yeast mitotic spindle experimental fluorescence image (kinetochore-associated fluorescence, green; spindle pole body fluorescence, red). (b) Theoretical point-source fluorophores (32 green points, representing individual kinetochores, and 2 red points, representing the spindle pole bodies) along a 1500 nm length. The bright green pixels indicate the presence of multiple fluorophores within the pixel area. For simplicity, it was assumed that there are no fluorophores in out-of-focus focal planes. (c) Point-source fluorophores in (a) are convolved with the microscope PSF and noise is added. (d) The image in (b) has been deconvolved using the identical PSF. The image deconvolution process cannot resolve the individual point-source fluorophores and tends to generate fluorescent “clusters” in the periphery which are artifacts of deconvolving noise

Full size image

In contrast, image deconvolution did not yield an accurate distribution of the kinetochores given the typical level of noise. Here, the deconvolution operation identified artificial clusters of green fluorescence as the underlying distribution of fluorescent kinetochores, both in the spindle as well as outside of the spindle (Fig. 1d). In this case, the Gaussian white noise in the background of the raw image results in “clusters” of fluorescent signal in the deconvolved image. The kinetochore example demonstrates two undesirable outcomes from deconvolution: (1) fluorophores are misplaced (inaccurate localization of green spots), and (2) noise is identified as signal (deconvolution of green background noise to produce artificial fluorescent clusters).

In the absence of noise, deconvolution is a simple linear operation that yields the exact fluorophore distribution in a single step. Noise is produced by photon statistical variance, which is defined as the square-root of the average integrated number of photons captured per detector picture element (pixel), and by detector read-out noise. For example, if the average number of photons per pixel in the background is 100, then the background photon noise is 10 photon counts per pixel. Similary, typical detector readout noise for cooled CCD cameras is equivalent to about 10 photon counts per pixel. In this case, the total noise per pixel for background fluorescence is the square root of (102 + 102) = ~14 photon counts (via the addition of variances). In addition, the signal itself also has photon noise, such that peak signal intensity fluctuates between image frames as the square root of the average number of photons. All of these noise sources confound the problem of separating meaningful signal from background signal. This uncertainty makes quantitative characterization of fluorescent protein locations via image deconvolution particularly problematic in images with a low signal-to-noise ratio (SNR), i.e., when the fluorescent probe markers are dim as compared to the image background noise.

To illustrate the effect of background noise on the accuracy of experimental deconvolution, we convolved a single fluorophore with a theoretical PSF, and then deconvolved the resulting image using a standard Wiener filter-based deconvolution algorithm, using the identical theoretical PSF (Fig. 2a). By adjusting background noise, we were able to achieve varying levels of success in identifying the simulated fluorophore after image deconvolution. Here, we define “success” as a result in which the brightest pixel in the deconvolved image is in the location of the original simulated fluorophore.

Figure 2
figure 2

High noise levels limit the utility of the image deconvolution method. (a1) A simulated point-source fluorophore has been convolved with a theoretical PSF (no background noise) to produce a 32 × 32 image having a single signal in the center of the field. (a2) Subsequent image deconvolution (by Wiener filtering-based deconvolution using the Matlab image processing toolbox) precisely resolves the spreading of light due to the PSF, and correctly identifies the fluorophore location to be at the center. (b1) A simulated point-source fluorophore has been convolved with a theoretical PSF, but noise has been added to the image such that the SNR = 8. (b2) In this case, subsequent image deconvolution is able to correctly resolve the fluorophore location. (b3) In another image with SNR = 8, image deconvolution is not able to separate the fluorophore from background noise and misidentifies the location of the point source. (c) The ability of Wiener-filter-based deconvolution to separate fluorophores from background noise decreases substantially with decreasing SNR. The quantitative relationship between the failure rate and the SNR depends upon the specifics of the problem, but generally failure rate increases with decreasing SNR

Full size image

If the SNR was large (>15), the deconvolution algorithm was nearly always able to separate signal from noise (Fig. 2a). In contrast, if the SNR was set equal to 8, it was just as likely for the deconvolution algorithm to successfully identify a fluorophore as it was for it to ignore the signal (Figs. 2b and 2c). As the SNR approaches 1, the deconvolution algorithm essentially picks the fluorophore location at random, and so yields no new information.

In the yeast mitotic spindle example (Fig. 1), although there are many fluorophores concentrated in a small area, experimental SNRs are ~3. In this case, the deconvolution algorithm is likely to confuse background noise with fluorophore signals, making quantitative spatial analysis of deconvolved images highly uncertain.

Model-Convolution in Quantitative Fluorescence Microscopy

We consider here an alternate method to deconvolution which we term “model-convolution.” This method enables digital fluorescence microscopy data to be interpreted in the context of a model with a measurable degree of confidence, when deconvolution is not successful because fluorophores are closely spaced and noise is significant (which is common for many live-cell imaging applications). With model-convolution, we use other structural information (from high-resolution electron micrographs, for example) to hypothesize an underlying distribution of photon counts from proteins or structures labeled with fluorophores.11 This 3D distribution is then convolved with the 3D distribution of the microscope PSF.8,9,25 The background fluorescence and noise typical of our microscope images is then added to obtain a simulated fluorescence image (Figs. 1b and 1c).25

To convolve a model for an underlying distribution of photon counts with the 3D distribution of the microscope PSF, the microscope PSF must first be defined. We have found that a reliable technique for defining a 3D PSF is to estimate the PSF experimentally by imaging sub-resolution fluorescent beads at different microscope focal planes.25,26 These images are then convolved with a 3-D model to account for the spreading of light through the microscope lens due to diffraction, as described below.

When defining a hypothesized spatial model for the underlying distribution of photon counts, the resolution of the model can, in theory, be infinitely small. However, the spatial model must ultimately be translated into a resolution that is consistent with the experimental imaging apparatus. For this reason, it is practical to define a “fine grid” array for the purpose of modeling the underlying fluorescent protein spatial distribution. Although the “fine grid” array used for modeling can be arbitrarily small, it is most accurate to define its dimensions such that it can be directly re-binned into the pixel size of the microscope used for imaging. For example, if the microscope pixel size is 66 × 66 nm, then an appropriate “fine grid” bin size may be 22 × 22 nm, so that nine “fine grid” elements can then be summed to correspond to the experimental pixel size.26

Once a hypothesized model has been defined and the corresponding “fine grid” array is re-binned such that each element of the final model grid is representative of an experimental microscope pixel size, then the model distribution is directly convolved with the experimentally measured 3D PSF. Here, the correct z-slice PSF is centered on each non-zero element in the binned model grid, and then the PSF is then convolved with the binned model grid. This can be accomplished through the use of commercially available software packages (e.g., MATLAB), which have built-in convolution commands.

After convolving the binned model grid with the experimentally measured PSF, then the final step in creating a model-convolved image is to add in the typical experimental noise and background fluorescence levels. The experimental background fluorescence and noise are established by measuring both the mean and the standard deviation of experimental fluorescence in an experimental image area that is far from the experimental signal. Then, this measured background fluorescence and noise can be applied to each pixel of the binned model grid, as follows:

$$ {\text{Bin}}_{\text{Final}} = {\text{Bin}}_{\text{convolved}} ( {\text{Signal}} - {\text{Bkrd)}} + {\text{Bkrd}} + {\text{rand}}*{\text{Noise}} $$
(1)

where BinFinal is the final model-convolved fluorescence intensity, calculated for each pixel of the image, Binconvolved is the binned grid element value after convolution with the PSF, Signal is the average measured experimental signal, Bkrd is the average measured experimental background, and Noise is the standard deviation of the background signal. Here, “rand” represents a Gaussian random number with mean = 0 and standard deviation = 1.

Quantitative statistical comparison of simulated to raw experimental microscope images then allows for quantitative evaluation of the theoretical fluorophore distribution. Previously, quantitative statistical comparison of simulated data to experimental data has been completed by calculating the sum-of-squares error for many individual simulations as compared to the overall average simulation curve, and then by calculating a p-value based on how the experimental sum-of-squares error compares to the simulation results (for detailed description of statistical methods, see Gardner et al.8 and Sprague et al.25).

By directly creating simulated fluorescence images from theoretical fluorophore arrangements, artifacts in deconvolution algorithms generated as a result of difficulties in distinguishing background noise from true signal are eliminated. Furthermore, in contrast to computational deconvolution, the final model-convolution simulated image is generated by incorporating known physical and molecular information about the system, thus simplifying interpretation of the results. Although deconvolution software iteratively seeks a fluorophore distribution that minimizes the sum-of-squares error, there may be other distributions with only slightly worse sum-of-squares error than the “best-fit” case, and which are quite different from the converged solution. The advantage of the model-convolution method is that it naturally incorporates the known physical and molecular constraints of the system (i.e., number of kinetochores, in the yeast mitosis case, etc.).

In situations with a limited number of fluorescent proteins and/or with multi-colored fluorescent proteins, 2D-Gaussian fitting and centroid tracking have been used effectively to achieve a spatial resolution beyond the Abbe limit.18,29 In this case, the model-convolution approach is helpful as a complement to standard tracking algorithms to establish the limits of detectability and accuracy. Validation of tracking algorithms is not routinely performed, but could be done readily with model-convolution.3,4

Applications for Model-Convolution

The model-convolution method is most useful as a technique for quantitative analysis and hypothesis testing with digital fluorescence microscopy. Cellular objects that are more closely spaced than half the wavelength of light used for imaging cannot be visually resolved as separate objects using single channel fluorescence microscopy. However, model-convolution can be used to accurately estimate the information content in fluorescence images. Clustering patterns of multiple fluorophores, changes in fluorophore locations over time and with genetic mutations, and recovery patterns of photobleached fluorophores can all be quantitatively analyzed using the model-convolution method as described above.

In our yeast mitotic spindle analysis, data from electron microscopy23,31 was used to program into a computer a three-dimensional geometric representation of a mitotic spindle. At metaphase in yeast, sister kinetochores stretch out their intervening centromere chromatin by pulling forces toward opposite poles generated by kinetochore attachment to a single microtubule. The 16 pairs of sister kinetochores, each tagged with Cse4-GFP, thus become centered on average near the spindle equator, although each sister kinetochore oscillates poleward and away from its pole, coupled to kinetochore microtubule plus-end depolymerization and polymerization at the kinetochore attachment sites. On average, spindle pole-to-pole length, measured via labeling with Spc29-CFP, is about 1500 nm in the yeast metaphase spindle, and each kinetochore microtubule is ~300 nm, such that sister kinetochore pairs are separated by ~900 nm. Theoretical numbers of kinetochore-associated fluorophores as well as the size and general organization of the mitotic spindle could be deduced from the three-dimensional ultrastructural analysis. Once the spindle geometry was defined based on electron microscopy, a computer simulation was run to test various models for the dynamics of fluorescently labeled proteins at kinetochores and the polymerization/depolymerization dynamics of kinetochore microtubules.9,25 In this way, a specific model for regulation of kinetochore microtubule polymerization/depolymerization was developed. This model relied on the regulation of microtubule dynamics via the physical separation of sister kinetochores. Thus, although individual sister kinetochore spacing cannot be resolved experimentally, the model-convolution method could be used to test models for the regulation of kinetochore microtubule dynamics via simulation of results from multiple experiments targeting both tubulin and kinetochore-associated proteins. In addition, statistical comparison of the model against the results of multiple experiments provided for quantitative validation of the model. Thus, a model for the dynamics of individual proteins in the spindle was developed via model-convolution, despite the inability to experimentally resolve individual fluorescent proteins via light microscopy.

The model-convolution method can also be used to design experiments for quantifying the physical separation distance between same-color fluorescent protein markers. As shown in Fig. 3a, simulated yeast mitotic spindle images for various sister kinetochore spacings can be generated via model-convolution (Fig. 3a). These images are similar to those expected for a mutant phenotype that modified kinetochore microtubule dynamics, resulting in altered sister kinetochore spacing. By quantitatively comparing the spacing of kinetochore-associated fluorescence clusters in simulated images, it is possible to predict the number of independent experimental images that would be required to make statistical conclusions about more subtle differences in sister kinetochore spacing between mutant and wild-type cells (Fig. 3b). For example, in order to spatially resolve differences in fluorescent cluster spacing of ~12 nm with a statistical significance of p < 0.01, ~50 different spindle experimental images would be required. Specifically, model-convolution simulations predict that if the distance between fluorescence cluster centroids is calculated for each unique spindle, and then this sampling of distances is statistically evaluated against the null hypothesis that the separation distance is equal to 0 nm, it would require 50 independent spindles to resolve a 12 nm centroid spacing with a statistical significance of p < 0.01.

Figure 3
figure 3

The model-convolution method as applied to experimental design. (a) Simulated images can be generated via model-convolution with varying mean distances between fluorescent protein clusters. In the upper images, mean distance between fluorescent protein clusters is ~600 nm, shown as raw simulated fluorophores and after model-convolution. In the lower images, mean distance between fluorescent protein clusters is ~800 nm. In this example, the difference in spacing is large enough to be distinguished readily upon visual examination. (b) For more subtle differences in fluorescent protein separation differences, statistical comparison of fluorescent protein localization is required. By generating simulated images of the yeast spindle example via model-convolution and performing a statistical comparison of these images, it is possible to predict the number of images required to achieve more subtle experimental spatial resolution. For example, to resolve a difference in fluorescent cluster spatial localization of ~12 nm, ~50 experimental images would be required

Full size image

Similar methods to the model-convolution technique have been employed in a variety of applications, including fluorescent speckling on microtubules,30 Ca2+ sparks in skeletal muscle,17 measurement of three-dimensional diffusion rates,19 and the sizing of small fluorescently labeled particles.7 In each of these cases, simulation of the fluorescence imaging process was employed to allow for testing of a theoretical model. A “distributed deconvolution” method similar to the model-deconvolution method was successfully employed by Littlefield and Fowler20 to measure thin filament lengths of myofibrils. Here, the authors detected length ranges within 40–100 nm by utilizing information from electron microscopy about the ultrastructural organization of actin and myosin filaments within the sarcomere repeat. This method relied on convolution of model data via a 1D Gaussian distribution.

The model-convolution method can also be used to aid in the interpretation of fluorescence images. As shown in Fig. 4a, two-dimensional fluorescence images can lead to misinterpretation of three-dimensional structures, such as branched actin filaments.14 From the simulated image, it is apparent that long side branches from the mother filament are nearly imperceptible, except at their base, when they project at various angles out of the focal plane. Assuming these filaments have the diameter of a single actin filament (7 nm), a theoretical structure in three dimensions is constructed and then model-convolution is used to generate a simulated fluorescence image (Fig. 4a). This allows one to assess the detectability of side branches as a function of branch angle and length.

Figure 4
figure 4

The model-convolution method as compared to the image deconvolution process. In the image deconvolution process, an experimental image is “deblurred” using the theoretical microscope PSF. With the model-convolution method, a theoretical fluorophore distribution is convolved with the microscope PSF and noise, and a simulated image is generated. Thus, the model-convolution method is essentially the inverse of the image deconvolution process. (a) The example shown is a computational model of Arp2/3-mediated actin filament branching in three dimensions based on experimental observations by Ichetovkin et al.14 The model results in a branched actin filament structure stemming from an initial nucleation site (1—blue arrow) and leading to a series of branches off the main filament. The model-convolution method is applied to create a theoretical microscope image at the focal plane of the main filament. Branches that are close to the focal plane of the microscope (2—orange arrow) are clearly visible in the simulated fluorescence image. Branches that project out of the focal plane (3—red arrow, and 4—white arrow) are less visible in the simulated image, indicating that the branching complexity and branch length distribution of the actin filament could be misinterpreted from experimental fluorescence images. Scale bar, 1000 nm. (b) The model-convolution approach to estimating microtubule curvature. A simulated microtubule is constructed with a known analytical function (Sine function on a 2 nm pixel grid), showing the true underlying relation of the curvature to the outer diameter. This simulated microtubule has curvature that would be at the high extreme of observed curvatures in living cells.24 The model-convolution operation is performed, and the resulting image is binned to the pixel size associated with a high NA lens and ccd detector (50 nm pixel size). The convolved image appears more highly curved than the underlying filament, and the digitization on the camera makes quantitative analysis of curvature prone to errors. Scale bar, 250 nm

Full size image

As another example, consider the analysis of microtubule bending in living cells. In this example, fluorescently tagged microtubules are ~25 nm in outer diameter, but fluorescent microscope images or images simulated via microtubule model-convolution yield a virtual microtubule image that is ~250 nm in diameter. The apparent 10-fold increase in filament thickness can lead to the perception that microtubule is more curved and under more mechanical stress than is actually the case. In addition, the presence of noise and the digitization inherent in CCD cameras can lead to substantial errors when quantitatively estimating the local curvature of a filament from a digital fluorescence image (Fig. 4b). In similar work by Janson and Dogterom, microtubule shape was digitized by convolution of microtubule image intensity profile with one period of a sine function to mimic the shadow-cast appearance of a microtubule16 imaged via differential interference contrast microscopy. This digitized data was then smoothed and used to analyze shape fluctuations on elongating microtubules.

Computational modeling is an emerging tool that can be used to better understand and predict complex cellular processes. By converting numerical data generated from computational models into simulated fluorescence images that can be directly compared to experimental images, the model-convolution method provides for a strong link between sophisticated simulation methods and experimental results. In this way, the results of computational simulations can be better understood and interpreted, regardless of the model complexity. In general, model-convolution facilitates the interpretation of the digital fluorescence microscopy data now being collected in large quantities.