Skip to main content
SpringerLink
Log in

Automating material image analysis for material discovery

  • Artificial Intelligence Prospective
  • Published:
MRS Communications Aims and scope Submit manuscript

Abstract

Advancements in temporal and spatial resolutions of microscopes promise to expand the frontiers of understanding in materials science. Imaging techniques produce images at a high-frame rate, streaming out a tremendous amount of data. Analysis of all these images is time-consuming and labor intensive, creating a bottleneck in material discovery that needs to be overcome. This paper summarizes recent progresses in machine learning and data science for expediting and automating material image analysis. The discussion covers both static image and dynamic image analyses, followed by remarks concerning ongoing efforts and future needs in automated image analysis that accelerates material discovery.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Figure 1
Figure 2
Figure 3
Figure 4
Figure 5

Similar content being viewed by others

References

  1. Basic Research Needs for Innovation and Discovery of Transformative Experimental Tools; available at http://science.energy.gov, 2017.

  2. A.V. Crewe: Scanning transmission electron microscopy. J. Microsc. 100, 247–259 (1974).

    Article  CAS  Google Scholar 

  3. S.M. Salapaka and M.V. Salapaka: Scanning probe microscopy. IEEE Control Syst. 28, 65–83 (2008).

    Article  Google Scholar 

  4. P. Abellan, B.L. Mehdi, L.R. Parent, M. Gu, C. Park, W. Xu, Y. Zhang, I. Arslan, J.-G. Zhang, and C.-M. Wang: Probing the degradation mechanisms in electrolyte solutions for Li-ion batteries by in situ transmission electron microscopy. Nano Lett. 14, 1293–1299 (2014).

    Article  CAS  Google Scholar 

  5. M.P. Chien, M.P. Thompson, C.V. Barback, T.H. Ku, D.J. Hall, and N.C. Gianneschi: Enzyme‐directed assembly of a nanoparticle probe in tumor tissue. Adv. Mater. 25, 3599–3604 (2013).

    Article  CAS  Google Scholar 

  6. J.E. Evans, K.L. Jungjohann, N.D. Browning, and I. Arslan: Controlled growth of nanoparticles from solution with in situ liquid transmission electron microscopy. Nano Lett. 11, 2809–2813 (2011).

    Article  CAS  Google Scholar 

  7. J.S. Kim, T. LaGrange, B.W. Reed, M.L. Taheri, M.R. Armstrong, W.E. King, N.D. Browning, and G.H. Campbell: Imaging of transient structures using nanosecond in situ TEM. Science 321, 1472–1475 (2008).

    Article  CAS  Google Scholar 

  8. T. LaGrange, G.H. Campbell, B. Reed, M. Taheri, J.B. Pesavento, J.S. Kim, and N.D. Browning: Nanosecond time-resolved investigations using the in situ of dynamic transmission electron microscope (DTEM). Ultramicroscopy 108, 1441–1449 (2008).

    Article  CAS  Google Scholar 

  9. T.J. Woehl, J.E. Evans, I. Arslan, W.D. Ristenpart, and N.D. Browning: Direct in situ determination of the mechanisms controlling nanoparticle nucleation and growth. ACS Nano 6, 8599–8610 (2012).

    Article  CAS  Google Scholar 

  10. T.J. Woehl, C. Park, J.E. Evans, I. Arslan, W.D. Ristenpart, and N.D. Browning: Direct observation of aggregative nanoparticle growth: kinetic modeling of the size distribution and growth rate. Nano Lett. 14, 373–378 (2013).

    Article  Google Scholar 

  11. J. P. Patterson, P. Abellan, M. S. Denny Jr., C. Park, N. D. Browning, S. M. Cohen, J. E. Evans, and N. C. Gianneschi: Observing the growth of metal–organic frameworks by in situ liquid cell transmission electron microscopy. J. Am. Chem. Soc. 137, 7322–7328 (2015).

    Article  CAS  Google Scholar 

  12. S. Jesse and S.V. Kalinin: Band excitation in scanning probe microscopy: sines of change. J. Phys. D: Appl. Phys. 44, 464006 (2011).

    Article  Google Scholar 

  13. B.J. Rodriguez, C. Callahan, S.V. Kalinin, and R. Proksch: Dual-frequency resonance-tracking atomic force microscopy. Nanotechnology 18, 475504 (2007).

    Article  Google Scholar 

  14. S.V. Kalinin, E. Strelcov, A. Belianinov, S. Somnath, R.K. Vasudevan, E.J. Lingerfelt, R.K. Archibald, C. Chen, R. Proksch, N. Laanait, and S. Jesse: Big, deep, and smart data in scanning probe microscopy. ACS Nano 10, 9068–9086 (2016).

    Article  CAS  Google Scholar 

  15. M.C. Roco: The Long View of Nanotechnology Development: The National Nanotechnology Initiative at 10 Years. Journal of Nanoparticles 13, 427–445 (2011).

    Article  Google Scholar 

  16. J. Canny: A computational approach to edge detection. IEEE Trans. Pattern Anal. Mach. Intell. 8, 679–698 (1986).

    Article  CAS  Google Scholar 

  17. N. Otsu: A threshold selection method from gray-level histograms. IEEE Trans. Syst. Man Cybern. 9, 62–66 (1979).

    Article  Google Scholar 

  18. X. Jiang and D. Mojon: Adaptive local thresholding by verification-based multithreshold probing with application to vessel detection in retinal images. IEEE Trans. Pattern Anal. Mach. Intell. 25, 131–137 (2003).

    Article  Google Scholar 

  19. G. Vo and C. Park: Robust regression for image binarization under heavy noises and nonuniform background. Pattern Recognit. 81, 224–239 (2018).

    Article  Google Scholar 

  20. C. Park, J.Z. Huang, D. Huitink, S. Kundu, B.K. Mallick, H. Liang, and Y. Ding: A multistage, semi-automated procedure for analyzing the morphology of nanoparticles. IIE Trans. 44, 507–522 (2012).

    Article  Google Scholar 

  21. C. Park, J.Z. Huang, J.X. Ji, and Y. Ding: Segmentation, inference and classification of partially overlapping nanoparticles. IEEE Trans. Pattern Anal. Mach. Intell. 35, 669–681 (2013).

    Google Scholar 

  22. S. Beucher and F. Meyer: The morphological approach to segmentation: the watershed transformation. Optical Engineering 34, 433–433 (1992).

    Google Scholar 

  23. Y. Qian, J.Z. Huang, X. Li, and Y. Ding: Robust nanoparticles detection from noisy background by fusing complementary image information. IEEE Trans. Image Process. 25, 5713–5726 (2016).

    Article  Google Scholar 

  24. S. Zafari, T. Eerola, J. Sampo, H. Kälviäinen, and H. Haario: Segmentation of overlapping elliptical objects in silhouette images. IEEE Trans. Image Process. 24, 5942–5952 (2015).

    Article  Google Scholar 

  25. X.-L. Meng, and D.B. Rubin: Maximum likelihood estimation via the ECM algorithm: a general framework. Biometrika 80, 267–278 (1993).

    Article  Google Scholar 

  26. P. Bubenik: Statistical topological data analysis using persistence landscapes. J. Mach. Learn. Res. 16, 77–102 (2015).

    Google Scholar 

  27. B.A. Konomi, S.S. Dhavala, J.Z. Huang, S. Kundu, D. Huitink, H. Liang, Y. Ding, and B.K. Mallick: Bayesian object classification of gold nanoparticles. Ann. Appl. Stat. 7, 640–668 (2013).

    Article  Google Scholar 

  28. J. Frank: Electron Tomography: Three-Dimensional Imaging with the Transmission Electron Microscope (New York, NY: Springer Science & Business Media, 2013).

    Google Scholar 

  29. C. Mu and C. Park: Optimal filtered backprojection for fast and accurate tomography reconstruction. Pattern Recognition Submitted (2019).

    Google Scholar 

  30. X. Li, A. Belianinov, O. Dyck, S. Jesse, and C. Park: Two-level structural sparsity regularization for identifying lattices and defects in noisy images. Ann. Appl. Stat. 12, 348–377 (2018).

    Article  Google Scholar 

  31. L. Dong, X. Li, Y. Qian, D. Yu, H. Zhang, Z. Zhang, and Y. Ding: Quantifying nanoparticle mixing state to account for both location and size effects. Technometrics 59, 391–403 (2017).

    Article  Google Scholar 

  32. A. Belianinov, Q. He, M. Kravchenko, S. Jesse, A. Borisevich, and S.V. Kalinin: Identification of phases, symmetries and defects through local crystallography. Nat. Commun. 6, 7801 (2015).

    Article  CAS  Google Scholar 

  33. D.S. Bright and E.B. Steel: Two‐dimensional top hat filter for extracting spots and spheres from digital images. J. Microsc. 146, 191–200 (1987).

    Article  Google Scholar 

  34. D. Sage, F.R. Neumann, F. Hediger, S.M. Gasser, and M. Unser: Automatic tracking of individual fluorescence particles: application to the study of chromosome dynamics. IEEE Trans. Image Process. 14, 1372–1383 (2005).

    Article  Google Scholar 

  35. J. Hughes, J. Fricks, and W. Hancock: Likelihood inference for particle location in fluorescence microscopy. Ann. Appl. Stat. 4, 830–848 (2010).

    Article  Google Scholar 

  36. N. Laanait, M. Ziatdinov, Q. He, and A. Borisevich: Identifying local structural states in atomic imaging by computer vision. Adv. Struct. Chem. Imaging 2, 14 (2017).

    Article  Google Scholar 

  37. B.D. Ripley: The second-order analysis of stationary point processes. J. Appl. Probab. 13, 255–266 (1976).

    Article  Google Scholar 

  38. X. Li, H. Zhang, J. Jin, D. Huang, X. Qi, Z. Zhang, and D. Yu: Quantifying dispersion of nanoparticles in polymer nanocomposites through transmission electron microscopy micrographs. J. Micro Nano-Manufacturing 2, 021008 (2014).

    Article  Google Scholar 

  39. N. De Jonge and F.M. Ross: Electron microscopy of specimens in liquid. Nat. Nanotechnol. 6, 695 (2011).

    Article  Google Scholar 

  40. S.V. Kalinin, B.G. Sumpter, and R.K. Archibald: Big-deep-smart data in imaging for guiding materials design. Nat. Mater. 14, 973 (2015).

    Article  CAS  Google Scholar 

  41. H. Zheng, Y.S. Meng, and Y. Zhu: Frontiers of in situ electron microscopy. MRS Bull. 40, 12–18 (2015).

    Article  Google Scholar 

  42. M. Grzelczak, J. Vermant, E.M. Furst, and L.M. Liz-Marzán: Directed selfassembly of nanoparticles. ACS Nano 4, 3591–3605 (2010).

    Article  CAS  Google Scholar 

  43. A. Mikhailov and G. Gundersen: Relationship between microtubule dynamics and lamellipodium formation revealed by direct imaging of microtubules in cells treated with nocodazole or taxol. Cytoskeleton 41, 325–340 (1998).

    Article  CAS  Google Scholar 

  44. L.G. Bergen and G.G. Borisy: Head-to-tail polymerization of microtubules in vitro. Electron microscope analysis of seeded assembly. J. Cell Biol. 84, 141–150 (1980).

    Article  CAS  Google Scholar 

  45. C. Park: Estimating multiple pathways of object growth using nonlongitudinal image data. Technometrics. 56, 186–199 (2014).

    Article  Google Scholar 

  46. C. Park and A.K. Shrivastava: Multimode geometric-profile monitoring with correlated image data and its application to nanoparticle selfassembly processes. J. Qual. Technol. 46, 216–233 (2014).

    Article  Google Scholar 

  47. C. Park, T.J. Woehl, J.E. Evans, and N.D. Browning: Minimum cost multiway data association for optimizing multitarget tracking of interacting objects. IEEE Trans. Pattern Anal. Mach. Intell. 37, 611–624 (2015).

    Article  Google Scholar 

  48. Y. Qian, J. Z. Huang; C. Park, and Y. Ding: Fast dynamic nonparametric distribution tracking in electron microscopic data. Ann. Appl. Stat. In press (2019).

    Google Scholar 

  49. Y. Qian, J.Z. Huang, and Y. Ding: Identifying multi-stage nanocrystal growth using in situ TEM video data. IISE Trans. 49, 532–543 (2017).

    Article  Google Scholar 

  50. H. Zheng, R.K. Smith, Y.-W. Jun, C. Kisielowski, U. Dahmen, and A.P. Alivisatos: Observation of single colloidal platinum nanocrystal growth trajectories. Science 324, 1309–1312 (2009).

    Article  CAS  Google Scholar 

  51. A. Rodriguez and E. Ter Horst: Bayesian dynamic density estimation. Bayesian Anal. 3, 339–365 (2008).

    Article  Google Scholar 

  52. R.H. Mena and M. Ruggiero: Dynamic density estimation with diffusive Dirichlet mixtures. Bernoulli. (Andover) 22, 901–926 (2016).

    Google Scholar 

  53. H. Jiang, S. Fels, and J. J. Little: In A linear programming approach for multiple object tracking, 2007 IEEE Conference on Computer Vision and Pattern Recognition, IEEE: 2007; pp. 1–8.

    Google Scholar 

  54. K. Okuma, A. Taleghani, N. De Freitas, J. J. Little, and D. G. Lowe: In A boosted particle filter: Multitarget detection and tracking, 2004 European Conference on Computer Vision, Springer, 2004; pp. 28–39.

    Google Scholar 

  55. H. Pirsiavash, D. Ramanan, and C. C. Fowlkes: In Globally-optimal greedy algorithms for tracking a variable number of objects, 2011 IEEE Conference on Computer Vision and Pattern Recognition, IEEE, 2011; pp. 1201–1208.

    Google Scholar 

  56. K. Jaqaman, D. Loerke, M. Mettlen, H. Kuwata, S. Grinstein, S.L. Schmid, and G. Danuser: Robust single-particle tracking in live-cell time-lapse sequences. Nat. Methods 5, 695 (2008).

    Article  CAS  Google Scholar 

  57. J. F. Henriques, R. Caseiro, and J. Batista: In Globally optimal solution to multi-object tracking with merged measurements, 2011 IEEE International Conference on Computer Vision, IEEE, 2011; pp. 2470–2477.

    Google Scholar 

  58. D.A. Welch, T.J. Woehl, C. Park, R. Faller, J.E. Evans, and N.D. Browning: Understanding the role of solvation forces on the preferential attachment of nanoparticles in liquid. ACS Nano 10, 181–187 (2015).

    Article  Google Scholar 

  59. A. Esmaieeli Sikaroudi, D.A. Welch, T.J. Woehl, R. Faller, J.E. Evans, N.D. Browning, and C. Park: Directional statistics of preferential orientations of two shapes in their aggregate and Its application to nanoparticle aggregation. Technometrics 60, 332–344 (2018).

    Article  Google Scholar 

  60. B.L. Mehdi, J. Qian, E. Nasybulin, C. Park, D.A. Welch, R. Faller, H. Mehta, W.A. Henderson, W. Xu, and C.M. Wang: Observation and quantification of nanoscale processes in lithium batteries by operando electrochemical (S) TEM. Nano Lett. 15, 2168–2173 (2015).

    Article  CAS  Google Scholar 

  61. M.A. Touve, C.A. Figg, D.B. Wright, C. Park, J. Cantlon, B.S. Sumerlin, and N.C. Gianneschi: Polymerization-induced self-assembly of micelles observed by liquid cell transmission electron microscopy. ACS Cent. Sci. 4, 543–547 (2018).

    Article  CAS  Google Scholar 

  62. A. Stevens, L. Luzi, H. Yang, L. Kovarik, B. Mehdi, A. Liyu, M. Gehm, and N. Browning: A sub-sampled approach to extremely low-dose STEM. Appl. Phys. Lett. 112, 043104 (2018).

    Article  Google Scholar 

  63. L. Kovarik, A. Stevens, A. Liyu, and N.D. Browning: Implementing an accurate and rapid sparse sampling approach for low-dose atomic resolution STEM imaging. Appl. Phys. Lett. 109, 164102 (2016).

    Article  Google Scholar 

  64. A. Stevens, L. Kovarik, P. Abellan, X. Yuan, L. Carin, and N.D. Browning: Applying compressive sensing to TEM video: a substantial frame rate increase on any camera. Adv. Struct. Chem. Imaging 1, 10 (2015).

    Article  Google Scholar 

  65. R. Castro, J. Haupt, and R. Nowak: In Compressed sensing vs. active learning, 2006 IEEE International Conference on Acoustics, Speech and Signal Processing, IEEE, 2006; p. III.

    Google Scholar 

  66. R. Edgeworth and R.G. Wilhelm: Adaptive sampling for coordinate metrology. Prec. Eng. 23, 144–154 (1999).

    Article  Google Scholar 

  67. C. Park and P. Qiu: Sequential Adaptive Design for Jump Regression Estimation. Submitted (IEEE Transactions on Pattern Analysis and Machine Intelligence 2019). Also available at https://arxiv.org/abs/1904.01648

    Google Scholar 

  68. A. H. Zewail and J. M. Thomas: 4D Electron Microscopy: Imaging in Space and Time. (Imperial College Press: London, 2009).

    Book  Google Scholar 

  69. S. Sreehari, S. Venkatakrishnan, K. L. Bouman, J. P. Simmons, L. F. Drummy, and C. A. Bouman: In Multi-resolution data fusion for superresolution electron microscopy, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2017; pp. 1084–1092.

    Google Scholar 

  70. H. Xia, Y. Ding, and B.K. Mallick: Bayesian hierarchical model for combining misaligned two-resolution metrology data. IIE Trans. 43, 242–258 (2011).

    Article  Google Scholar 

  71. A.A. Ezzat, A. Pourhabib, and Y. Ding: Sequential design for functional calibration of computer models. Technometrics. 60, 286–296 (2018).

    Article  Google Scholar 

  72. A. Pourhabib, J.Z. Huang, K. Wang, C. Zhang, B. Wang, and Y. Ding: Modulus prediction of buckypaper based on multi-fidelity analysis involving latent variables. IIE Trans. 47, 141–152 (2015).

    Article  Google Scholar 

Download references

Acknowledgments

We acknowledge support for this work from the AFOSR (FA9550-18-1-0144) and the Oak Ridge National Lab (4000152630).

Author information

Authors and Affiliations

  1. Department of Industrial and Manufacturing Engineering, Florida State University, Tallahassee, FL, 32310, USA

    Chiwoo Park

  2. Department of Industrial and Systems Engineering, Texas A&M University, 3131 TAMU, College Station, TX, 77843, USA

    Yu Ding

Authors
  1. Chiwoo Park
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yu Ding
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yu Ding.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Park, C., Ding, Y. Automating material image analysis for material discovery. MRS Communications 9, 545–555 (2019). https://doi.org/10.1557/mrc.2019.48

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1557/mrc.2019.48

Search

Navigation