Abstract
Single-beam scanning electron microscopes (SEM) are widely used to acquire massive datasets for biomedical study, material analysis, and fabrication inspection. Datasets are typically acquired with uniform acquisition: applying the electron beam with the same power and duration to all image pixels, even if there is great variety in the pixels’ importance for eventual use. Many SEMs are now able to move the beam to any pixel in the field of view without delay, enabling them, in principle, to invest their time budget more effectively with non-uniform imaging.
In this paper, we show how to use deep learning to accelerate and optimize single-beam SEM acquisition of images. Our algorithm rapidly collects an information-lossy image (e.g. low resolution) and then applies a novel learning method to identify a small subset of pixels to be collected at higher resolution based on a trade-off between the saliency and spatial diversity. We demonstrate the efficacy of this novel technique for active acquisition by speeding up the task of collecting connectomic datasets for neurobiology by up to an order of magnitude. Code is available at https://github.com/lumi9587/learning-guided-SEM.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Anderson, H.S., Ilic-Helms, J., Rohrer, B., Wheeler, J., Larson, K.: Sparse imaging for fast electron microscopy. In: Computational Imaging XI, vol. 8657, p. 86570C. International Society for Optics and Photonics (2013)
Buchholz, T.O., Krull, A., Shahidi, R., Pigino, G., Jékely, G., Jug, F.: Content-aware image restoration for electron microscopy. Meth. Cell Biol. 152, 277–289 (2019)
Dahmen, T.: Feature adaptive sampling for scanning electron microscopy. Sci. Rep. 6, 25350 (2016)
Eberle, A., Mikula, S., Schalek, R., Lichtman, J., Tate, M.K., Zeidler, D.: High-resolution, high-throughput imaging with a multibeam scanning electron microscope. J. Micros. 259(2), 114–120 (2015)
Eldar, Y.C., Kutyniok, G.: Compressed Sensing: Theory and Applications. Cambridge University Press, Cambridge (2012)
Fang, L., et al.: Deep learning-based point-scanning super-resolution imaging. bioRxiv, p. 740548 (2019)
Flegler, S.L., Flegler, S.L.: Scanning & Transmission Electron Microscopy. Oxford University Press, Oxford (1997)
Gan, L.: Block compressed sensing of natural images. In: 2007 15th International Conference on Digital Signal Processing, pp. 403–406. IEEE (2007)
Helmstaedter, M., Briggman, K.L., Denk, W.: High-accuracy neurite reconstruction for high-throughput neuroanatomy. Nat. Neurosci. 14(8), 1081–1088 (2011)
Ilg, E.: Uncertainty estimates and multi-hypotheses networks for optical flow. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11211, pp. 677–693. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01234-2_40
Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1125–1134 (2017)
Januszewski, M.: High-precision automated reconstruction of neurons with flood-filling networks. Nat. Meth. 15, 605–610 (2018)
Jarrell, T.A.: The connectome of a decision-making neural network. Science 337(6093), 437–444 (2012)
Kasthuri, N., et al.: Saturated reconstruction of a volume of neocortex. Cell 162(3), 648–661 (2015)
Kulesza, A., Taskar, B., et al.: Determinantal point processes for machine learning. Found. Trends Mach. Learn. 5(2–3), 123–286 (2012)
Ledig, C., et al.: Photo-realistic single image super-resolution using a generative adversarial network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4681–4690 (2017)
Lichtman, J.W., Pfister, H., Shavit, N.: The big data challenges of connectomics. Nat. Neurosci. 17(11), 1448–1454 (2014)
Meirovitch, Y., Mi, L., Saribekyan, H., Matveev, A., Rolnick, D., Shavit, N.: Cross-classification clustering: an efficient multi-object tracking technique for 3-D instance segmentation in connectomics. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8425–8435 (2019)
Mi, L., Wang, H., Tian, Y., Shavit, N.: Training-free uncertainty estimation for neural networks. arXiv preprint arXiv:1910.04858 (2019)
Mohammed, A.: Scanning electron microscopy (SEM): a review (2018)
Newell, T., Tillotson, B., Pearl, H., Miller, A.: Detection of electrical defects with semvision in semiconductor production mode manufacturing. In: 2016 27th Annual SEMI Advanced Semiconductor Manufacturing Conference (ASMC), pp. 151–156. IEEE (2016)
Pandey, K., Setua, D., Mathur, G.: Material behaviour: fracture topography of rubber surfaces: an SEM study. Polym. Testing 22(3), 353–359 (2003)
Potocek, P., Trampert, P., Peemen, M., Schoenmakers, R., Dahmen, T.: Sparse scanning electron microscopy data acquisition and deep neural networks for automated segmentation in connectomics. Microsc. Microanal. 26, 403–412 (2020)
Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28
Thorpe, S., Fize, D., Marlot, C.: Speed of processing in the human visual system. Nature 381(6582), 520 (1996)
Wang, G., Li, W., Aertsen, M., Deprest, J., Ourselin, S., Vercauteren, T.: Test-time augmentation with uncertainty estimation for deep learning-based medical image segmentation. arXiv preprint arXiv:1807.07356 (2018)
Wang, H., et al.: Deep learning enables cross-modality super-resolution in fluorescence microscopy. Nat. Meth. 16, 103–110 (2019)
Weigert, M., et al.: Content-aware image restoration: pushing the limits of fluorescence microscopy. Nat. Meth. 15(12), 1090 (2018)
Yan, G., et al.: Network control principles predict neuron function in the caenorhabditis elegans connectome. Nature 550(7677), 519 (2017)
Acknowledgement
We would like to thank Remco Schoenmakers and Pengfei Guo for insightful comments and suggestions. This research was supported by the National Science Foundation (NSF) under grants IIS-1607189, CCF-1563880, IOS-1452593 and NSF 1806818.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Mi, L. et al. (2020). Learning Guided Electron Microscopy with Active Acquisition. In: Martel, A.L., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2020. MICCAI 2020. Lecture Notes in Computer Science(), vol 12265. Springer, Cham. https://doi.org/10.1007/978-3-030-59722-1_8
Download citation
DOI: https://doi.org/10.1007/978-3-030-59722-1_8
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-59721-4
Online ISBN: 978-3-030-59722-1
eBook Packages: Computer ScienceComputer Science (R0)
