Automated analysis of evolving interfaces during in situ electron microscopy
In situ electron microscopy allows one to monitor dynamical processes at high spatial and temporal resolution. This produces large quantities of data, and hence automated image processing algorithms are needed to extract useful quantitative measures of the observed phenomena. In this work, we outline an image processing workflow for the analysis of evolving interfaces imaged during liquid cell electron microscopy. As examples, we show metal electrodeposition at electrode surfaces; beam-induced nanocrystal formation and dissolution; and beam-induced bubble nucleation, growth, and migration. These experiments are used to demonstrate a fully automated workflow for the extraction of, among other things, interface position, roughness, lateral wavelength, local normal velocity, and the projected area of the evolving phase as functions of time. The relevant algorithms have been implemented in Mathematica and are available online.
Keywords(10 max): In situ Electron microscopy Image analysis Liquid cell electron microscopy Interface tracking Growth dynamics Processes Edge detection
Detailed understanding of the evolution of interfaces is of both scientific and practical interest in many disciplines. Scientists wish to identify unifying principles of pattern formation in processes such as solidification and aggregation . Engineers are interested in controlling the evolution of interfaces and their morphologies. For example, morphological instabilities, particularly dendrite formation, can cause catastrophic failure in rechargeable batteries or lower the quality of electroplated coatings, yet they may be useful in forming porous deposits such as porous electrodes and catalytic surfaces; real-time observations provide essential information on the process by which they form [2, 3, 4, 5, 6]. Nanocrystal growth is another example where morphology control enables tuning of mechanical, electrical, and optical properties [7, 8, 9, 10]. Liquid cell electron microscopy [11, 12] can address such problems. It allows us to image, in real time and with nanoscale resolution, processes such as the evolution of the growth front during electrochemical deposition and etching [13, 14, 15, 16, 17, 18, 19], nanocrystal formation and dissolution [9, 20, 21, 22, 23], and bubble nucleation, growth, and migration [11, 12, 14, 15, 16, 20, 21, 24] to provide insights into the mechanisms controlling morphology and mass transport.
Making the best use of the information obtained during liquid cell experiments requires quantitative measurements from each frame in a video sequence as a phenomenon of interest takes place. In particular, tracking the position of interfaces between different materials is a common first step. For example, Woehl et al.  studied the nucleation and growth of silver nanocrystals utilizing image processing to obtain particle count and radius as functions of time and beam conditions, and Ievlev et al.  used interface detection algorithms to investigate platinum nanocrystal growth properties via local growth velocity. We have therefore developed a general use interface-tracking algorithm based on image analysis needs over several different liquid cell experiments. Here we demonstrate its use, extract several quantitative measures of dynamical processes, and outline the algorithm details for easy adaptation of our open source software  to other experimental systems. Our workflow is illustrated with experimental data from three liquid cell experiments: copper electrodeposition [18, 19], electron beam-induced formation of gold nanoprisms [9, 21], and bubble nucleation and growth . For copper electrodeposition from an acidified copper solution [18, 19], we extract the growth front location as a function of time. The data are then used to calculate the projected area and mass of the deposit, the root mean square (RMS) roughness of the interface to identify the growth habit, the local normal velocity distribution (akin to current density) to provide the relationship between geometry and point-wise growth rate, and the wavelengths associated with interface morphology, as their control is important for structured surface manufacturing. For growth and dissolution of single-crystal Au nanoparticles mediated by the electron beam [9, 20, 21], the algorithm adjusts for particle translation and extracts the particle orientation to properly determine growth kinetics. Finally, for nanobubble nucleation, growth, and migration [24, 26], we extract bubble radius and position as functions of time. The automation of the process allows for analysis of datasets with large numbers of frames.
Several processes were combined to enable unsupervised, non-parametric image processing, and we outline the individual steps below. By non-parametric, we mean that frame-specific parameters are automatically calculated without user input. The data used for illustration were obtained as described in the experimental methods section. The open source code has been well-documented to make it accessible to new Mathematica users. A video walkthrough of the code is available in the supplement (Additional file 1: Video S1) and on YouTube .
Raw data pre-preparation
Image preparation—noise filtering and background subtraction
To minimize damage caused by the electron beam, it is important to operate at the minimum dose rate required to form a usable image [31, 32]. This results in images with a relatively low signal-to-noise-ratio (SNR). Since the Rose criterion requires the SNR to be about 5 for an image to be detectable by the human eye [33, 34], it is useful to take precautions to reduce the apparent noise in each frame of the video sequence while preserving edges. We achieve this reduction through background subtraction and the application of a total variation filter (TV filter). We illustrate this technique using experimental data from beam-induced gold nanocrystal formation [9, 21].
Thresholding and edge detection
Once noise is filtered and contrast enhanced, edge detection can be performed. Edge detection works best on a binary image. Hence, thresholding is first performed. The standard Otsu’s method  is utilized to convert the grayscale image into one with only two values (additional gray levels can be included when needed). This process once again casts the image analysis as an optimization problem, where now we assume that pixels can be classified into two classes, the foreground and background, with maximized separability. That is, the pixels are assumed to have a bi-modal histogram and the algorithm tests all possible threshold values, where the chosen threshold value is the one where the sum of the variances of the two classes is minimal. Figure 2g shows a binarized foreground image that was TV filtered prior to thresholding.
Once the binary image is generated, one can find the edges of the object. The edges are a direct measure of the interface as projected from the image plane. A Canny style edge detector  is extremely robust for images of the type depicted in Fig. 2g. Edges are defined as local maxima in the first-order directional Gaussian gradient of the intensity function. Since the image has already been denoised and binarized, the edges are extracted with great fidelity as shown in Fig. 2h. Had the image not been pre-processed with the TV filter and background subtraction, many false positives would have been identified by the edge detection (Fig. 2i).
Translation correction and object centroids
Often, image sequences contain drift and/or migration of the imaged object. When examining the growth of the object, it is convenient to fix the location of its center of mass in all frames. For example, Additional file 2: Video S2 shows the growth of a gold crystal. During the video, due to drift adjustments of the beam location, the relative position of the crystal has changed. Here, we present a means for the translating a single object to fix the location of its center of mass in all the frames. As part of this process, we track the location of the object’s centroid as a function of time. The code can be extended to systems of multiple dynamic objects via particle tracking algorithms such as the methods developed by Crocker and Grier  that are available as open source code for several computation platforms. This software relates the positions of entities between frames to statistically identify the displacement of individual entities and establish their track (path in time). For simplicity, here, we will only extract the positions of particles subject to translation.
Growth front tracking during electrodeposition
The first application of our image processing algorithm is to extract the growth front morphology during copper electrodeposition. Copper electrodeposition experiments were carried out with our custom-made liquid cell, the nanoaquarium , operating in a three-terminal configuration with Pt electrodes controlled by a Gamry potentiostat. The liquid cell was filled with an aqueous solution of 0.1 M CuSO4 + 0.18 M H2SO4, prepared in doubly deionized water and not deaerated before use. The interface morphology evolution was recorded using bright-field imaging conditions during galvanostatic (constant current) deposition. Data were recorded at video rate (30 frames per second) in a Hitachi H-9000 TEM at 300 kV. In these types of experiments, liquid cell electron microscopy provides the surface evolution in real time and with nanoscale resolution [11, 12, 14, 19] and the aim of the quantitative analysis is to obtain insight into the mechanisms leading to stable and unstable growth fronts under different conditions.
An alternative method exploits the fact that the shortest distance from a point to a surface/curve lies along the normal to the curve. To construct a normal to the surface St that intersects point j on surface St+Δt, we seek point i * on St such that the distance between points i and j is minimized. Figure 4c illustrates this process. In fact, Mathematica has built-in routines to quickly find which point from a point cloud (ordered or not) is closest to a point of interest. This proved to be a very effective and fast means for the extraction of point-wise normal velocity, especially in the presence of noise.
We applied the velocity routine to the evolving interface of Additional file 3: Video S3. Figure 3e shows the resulting “heat map” (contour plot) of the normal speed in space (along the interface) as a function of time. Initially, we see uniform distribution of speeds and therefore uniform current density. As the roughness develops due to natural fluctuations, the current density becomes less uniform locally, leading to continued amplification of the roughness.
Facet-dependent growth and dissolution rates in nanoparticles
Size and morphology control of nanocrystals is essential to tailor the physical and chemical properties of the resulting material [7, 8]. Liquid phase synthesis is a convenient, low-cost and versatile technique for fabricating well-controlled nanocrystals, since solution chemistry can be tuned to give specific morphology. A fundamental understanding of the driving force and principles that govern morphologies during nucleation and growth is essential to optimize existing synthesis methods and to discover novel methods. Experiments on the dissolution and growth of gold nanorods were carried out with the nanoaquarium in an FEI Quanta 600 FEG Mark II SEM with a transmission detector. The aqueous solution contained gold nanorods  with a trace amounts of the surfactant cetrimonium bromide (CTAB) and HAuCl4 at pH ~7. The devices were filled under atmospheric conditions and the solution was not deaerated. Production of radiolytic species by the electron beam, as described in Schneider et al. , causes the Au nanorods to either grow or dissolve. During dissolution, we hypothesize that the radicals produced during water irradiation react with the CTAB, whose role is to stabilize the nanorods; in the absence of CTAB, nanorods are known to be unstable and dissolve [49, 50]. Analysis of morphology and growth dynamics at specific facets provides information about the atomic level processes that control shape .
Bubble nucleation, growth, and migration
The dynamics of gas–liquid interfaces plays an important role in many physical processes such as high efficiency phase change heat transfer. In recent years, the existence and stability of nanobubbles have been debated [55, 56, 57, 58]. Recently, atomic force microscopy (AFM) measurements have confirmed the existence and unraveled the geometry of nanobubbles [55, 56, 58, 59]. However, such measurements have limited spatial resolution and there is an additional concern that the mechanical perturbation applied by the cantilever is too invasive. Electron microscopy imaging of nanobubbles provides information that complements AFM measurements.
Here, we consider the motion of radiolytically formed nanobubbles confined in a tapered conduit. Given the bubbles’ size, both the capillary and Bond numbers are essentially zero. In this fluid dynamical regime, motion and geometry of the nanobubbles are governed by three-phase contact line wetting and de-wetting. Identifying the interfaces of single bubbles and tracking their trajectories is essential in obtaining the quantitative information needed to model contact line dynamics. In this case, both the contact line velocities and trajectories of the centroids are of interest.
Data processing and analysis tutorial
For each of the three examples given above, i.e., growth front tracking during electrochemical deposition, nanoparticle dissolution, and bubble motion, an overview is described in the supplemental Additional file 6: Code S1. This is available on GitHub  with an accompanying video tutorial Additional file 1: Video S1. Fully automatic and unsupervised analysis of the three sample datasets presented in this paper is available in the supplemental Additional file 7: Code S2, Additional file 8: Code S3, Additional file 9: Code S4, also maintained on GitHub .
Liquid cell electron microscopy is a powerful technique with many applications to dynamic processes in liquid media. The extraction of quantitative measures is essential for proper interpretation of the data. However, typical experiments generate very large quantities of data, which can be a challenge to analyze quantitatively. We have presented a workflow that is suited to analyze typical data of the type collected during liquid cell electron microscopy. We chose Mathematica as the platform to implement the various procedures because of its ease of use, availability of numerous built-in algorithms for image processing, automation, and flexibility. We hope that the tools presented here and the tutorials provided in the supplement will be useful to the microscopy community for diverse applications of data analysis for dynamical processes both in liquid cell experiments and for in situ microscopy in general.
NMS, FMR, and HHB conceived the project and all interpreted the results and wrote the manuscript. NMS, JHP, MMN, and FMR designed and carried out the experiments. NMS implemented image processing schemes and conducted data analysis. All authors read and approved the final manuscript.
The nanoaquarium fabrication was carried out at the Cornell NanoScale Facility (NSF Grant ECS-0335765), a member of the National Nanotechnology Infrastructure Network. Electron microscopy was carried out at the Penn Regional Nanotechnology Facility and the IBM T. J. Watson Research Center with the valuable assistance of Dr. Joseph M. Grogan and Mr. Peter Szczesniak of UPenn and Dr. Mark C. Reuter and Mr. Arthur Ellis of IBM. Gold nanorods were generously provided by Dr. Christopher Murray. The work was supported, in part, by the National Science Foundation Grants 1129722 and 1066573 to the University of Pennsylvania, and 1310639 to the University of California Los Angeles.
The authors declare that they have no competing interests.
- 3.Sun. M., Liao. H-G., Niu K., Zheng. H.: Structural and morphological evolution of lead dendrites during electrochemical migration. Sci Reports 3 (2013). doi: 10.1038/srep03227
- 6.Zeng. Z., Liang. W-I., Liao. H-G., et al.: visualization of electrode-electrolyte interfaces in lipf6/ec/dec electrolyte for lithium ion batteries via in-situ TEM. Nano letters 140120111747008 (2014). doi: 10.1021/nl403922u
- 9.Park. JH., Schneider. NM., Grogan JM., et al.: control of electron beam-induced au nanocrystal growth kinetics through solution chemistry. Nano letters 150729101858009 (2015). doi: 10.1021/acs.nanolett.5b01677
- 12.Grogan, J.M., Schneider, N.M., Ross, F.M., Bau, H.H.: The nanoaquarium: a new paradigm in electron microscopy. J Indian Inst Sci 92, 295–308 (2012)Google Scholar
- 13.Mehdi. BL., Qian. J., Nasybulin. E., et al.: Observation and quantification of nanoscale processes in lithium batteries by operando electrochemical (s)TEM. Nano letters 150225082830003 (2015). doi: 10.1021/acs.nanolett.5b00175
- 17.Leenheer. AJ., Sullivan. JP., Shaw. MJ., Harris. CT.: A sealed liquid cell for in situ transmission electron microscopy of controlled electrochemical processes. Microelectromechanical Systems J 1–1. doi: 10.1109/JMEMS.2014.2380771
- 24.Norton. M., Park. J., Kodambaka. S., et al.: Dynamics of Sub-micron bubbles growing in a wedge in the low capillary number regime. bulletin of the american physical society 59, 20 (2014)Google Scholar
- 25.Schneider NM NMSchneider/Quantitative-Image-Analysis. In: github.com. https://github.com/NMSchneider/Quantitative-Image-Analysis. Accessed 13 Feb 2015
- 27.Schneider. NM.: Video S1 Code S1 walk through. Online video clip. YouTube. YouTube, 9 Aug 2015. Web. 24 Jan 2016. (2015)Google Scholar
- 28.Lee. A.: Virtualdub. http://www.virtualdub.org (2015). Accessed 24 Jan 2015
- 29.Rasband, W.S., ImageJ, U. S.: National Institutes of Health, Maryland http://imagej.nih.gov/ij/ (1997–2015). Accessed 24 Jan 2015
- 30.Schneider. NM.: liquid cell electron microscopy with the nanoaquarium: radition and electrochemistry. 1–222 (2015)Google Scholar
- 37.Piccardi. M.: Background subtraction techniques: a review. ICSMC-04 4. 4, 3099–3104. doi: 10.1109/ICSMC.2004.1400815
- 38.Otsu. N.: A threshold selection method from gray-level histograms. systems, man and cybernetics, IEEE transactions on 9, 62–66 (1979). doi: 10.1109/TSMC.1979.4310076
- 39.Canny. J.: A computational approach to edge detection. pattern analysis and machine intelligence, IEEE Transactions on 679–698. (1986). doi: 10.1109/TPAMI.1986.4767851
- 46.Grinfeld. P.: Introduction to tensor analysis and the calculus of moving surfaces. (2013). doi: 10.1007/978-1-4614-7867-6
- 52.Toussaint. GT.: Solving geometric problems with the rotating calipers. Proc IEEE Melecon. (1983)Google Scholar
- 55.Tyrrell. JWG., Attard. P.: Atomic force microscope images of nanobubbles on a hydrophobic surface and corresponding force-separation data. 18, 160–167 (2002). doi: 10.1021/la0111957
- 57.Attard, P., Moody. MP., Tyrrell. J.: Nanobubbles: the big picture. Physica A: statistical mechanics and its ….. (2002). doi: 10.1016/S0378-4371(02)01191-3
- 58.Attard, P.: Nanobubbles and the hydrophobic attraction. 104, 75–91 (2003). doi: 10.1016/S0001-8686(03)00037-X
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.