Two Stream Active Query Suggestion for Active Learning in Connectomics

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12363)


For large-scale vision tasks in biomedical images, the labeled data is often limited to train effective deep models. Active learning is a common solution, where a query suggestion method selects representative unlabeled samples for annotation, and the new labels are used to improve the base model. However, most query suggestion models optimize their learnable parameters only on the limited labeled data and consequently become less effective for the more challenging unlabeled data. To tackle this, we propose a two-stream active query suggestion approach. In addition to the supervised feature extractor, we introduce an unsupervised one optimized on all raw images to capture diverse image features, which can later be improved by fine-tuning on new labels. As a use case, we build an end-to-end active learning framework with our query suggestion method for 3D synapse detection and mitochondria segmentation in connectomics. With the framework, we curate, to our best knowledge, the largest connectomics dataset with dense synapses and mitochondria annotation. On this new dataset, our method outperforms previous state-of-the-art methods by 3.1% for synapse and 3.8% for mitochondria in terms of region-of-interest proposal accuracy. We also apply our method to image classification, where it outperforms previous approaches on CIFAR-10 under the same limited annotation budget. The project page is


Active learning Connectomics Object detection Semantic segmentation Image classification 



This work has been partially supported by NSF award IIS-1835231 and NIH award 5U54CA225088-03.

Supplementary material

504473_1_En_7_MOESM1_ESM.pdf (4.1 mb)
Supplementary material 1 (pdf 4156 KB)


  1. 1.
    Abramson, Y., Freund, Y.: Active learning for visual object detection (2006)Google Scholar
  2. 2.
    Becker, C., Ali, K., Knott, G., Fua, P.: Learning context cues for synapse segmentation. IEEE TMI 32, 1864–1877 (2013)Google Scholar
  3. 3.
    Belkin, M., Niyogi, P.: Using manifold stucture for partially labeled classification. In: NIPS (2003)Google Scholar
  4. 4.
    Bietti, A.: Active learning for object detection on satellite images. Technical report, Caltech (2012)Google Scholar
  5. 5.
    Buhmann, J., et al.: Synaptic partner prediction from point annotations in insect brains. arXiv preprint arXiv:1806.08205 (2018)
  6. 6.
    Cheng, H.C., Varshney, A.: Volume segmentation using convolutional neural networks with limited training data. In: ICIP (2017)Google Scholar
  7. 7.
    Dorkenwald, S., et al.: Automated synaptic connectivity inference for volume electron microscopy. Nat. Methods 14(4), 435 (2017)CrossRefGoogle Scholar
  8. 8.
    Ducoffe, M., Precioso, F.: Adversarial active learning for deep networks: a margin based approach. In: ICML (2018)Google Scholar
  9. 9.
    Dutt Jain, S., Grauman, K.: Active image segmentation propagation. In: CVPR (2016)Google Scholar
  10. 10.
    Freytag, A., Rodner, E., Denzler, J.: Selecting influential examples: active learning with expected model output changes. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8692, pp. 562–577. Springer, Cham (2014). Scholar
  11. 11.
    Funke, J., Saalfeld, S., Bock, D., Turaga, S., Perlman, E.: Circuit reconstruction from electron microscopy images (2016).
  12. 12.
    He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask R-CNN. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2961–2969 (2017)Google Scholar
  13. 13.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR (2016)Google Scholar
  14. 14.
    Heinrich, L., Funke, J., Pape, C., Nunez-Iglesias, J., Saalfeld, S.: Synaptic cleft segmentation in non-isotropic volume electron microscopy of the complete Drosophila brain. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11071, pp. 317–325. Springer, Cham (2018). Scholar
  15. 15.
    Hoffer, E., Ailon, N.: Deep metric learning using triplet network. In: International Workshop on Similarity-Based Pattern Recognition (2015)Google Scholar
  16. 16.
    Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: CVPR (2018)Google Scholar
  17. 17.
    Huang, G.B., Plaza, S.: Identifying synapses using deep and wide multiscale recursive networks. arXiv preprint arXiv:1409.1789 (2014)
  18. 18.
    Huang, G.B., Scheffer, L.K., Plaza, S.M.: Fully-automatic synapse prediction and validation on a large data set. arXiv preprint arXiv:1604.03075 (2016)
  19. 19.
    Jagadeesh, V., Anderson, J., Jones, B., Marc, R., Fisher, S., Manjunath, B.: Synapse classification and localization in electron micrographs. Pattern Recogn. Lett. 43, 17–24 (2014)CrossRefGoogle Scholar
  20. 20.
    Kao, C.-C., Lee, T.-Y., Sen, P., Liu, M.-Y.: Localization-aware active learning for object detection. In: Jawahar, C.V., Li, H., Mori, G., Schindler, K. (eds.) ACCV 2018. LNCS, vol. 11366, pp. 506–522. Springer, Cham (2019). Scholar
  21. 21.
    Kasthuri, N., et al.: Saturated reconstruction of a volume of neocortex. Cell 162(3), 648–661 (2015)CrossRefGoogle Scholar
  22. 22.
    Kingma, D.P., Welling, M.: Auto-encoding variational bayes. ICLR (2013)Google Scholar
  23. 23.
    Kreshuk, A., Funke, J., Cardona, A., Hamprecht, F.A.: Who is talking to whom: synaptic partner detection in anisotropic volumes of insect brain. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9349, pp. 661–668. Springer, Cham (2015). Scholar
  24. 24.
    Kreshuk, A., Koethe, U., Pax, E., Bock, D.D., Hamprecht, F.A.: Automated detection of synapses in serial section transmission electron microscopy image stacks. PLoS One 9(2), e87351 (2014)CrossRefGoogle Scholar
  25. 25.
    Kreshuk, A., et al.: Automated detection and segmentation of synaptic contacts in nearly isotropic serial electron microscopy images. PLoS One 6(10), e24899 (2011)CrossRefGoogle Scholar
  26. 26.
    Lichtman, J.W., Sanes, J.R.: Ome sweet ome: what can the genome tell us about the connectome? Curr. Opin. Neurobiol. 18(3), 346–353 (2008)CrossRefGoogle Scholar
  27. 27.
    Lucchi, A., Li, Y., Fua, P.: Learning for structured prediction using approximate subgradient descent with working sets. In: CVPR (2013)Google Scholar
  28. 28.
    Lucchi, A., Li, Y., Smith, K., Fua, P.: Structured image segmentation using kernelized features. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7573, pp. 400–413. Springer, Heidelberg (2012). Scholar
  29. 29.
    Lucchi, A., et al.: Learning structured models for segmentation of 2D and 3D imagery. IEEE TMI 34, 1096–1110 (2015)Google Scholar
  30. 30.
    Lucchi, A., Smith, K., Achanta, R., Knott, G., Fua, P.: Supervoxel-based segmentation of mitochondria in EM image stacks with learned shape features. IEEE TMI 31, 474–486 (2012)Google Scholar
  31. 31.
    Morgan, J.L., Lichtman, J.W.: Why not connectomics? Nat. Methods 10(6), 494 (2013)CrossRefGoogle Scholar
  32. 32.
    Narasimha, R., Ouyang, H., Gray, A., McLaughlin, S.W., Subramaniam, S.: Automatic joint classification and segmentation of whole cell 3D images. Pattern Recogn. 42, 1067–1079 (2009)CrossRefGoogle Scholar
  33. 33.
    Oztel, I., Yolcu, G., Ersoy, I., White, T., Bunyak, F.: Mitochondria segmentation in electron microscopy volumes using deep convolutional neural network. In: Bioinformatics and Biomedicine (2017)Google Scholar
  34. 34.
    Parag, T., et al.: Detecting synapse location and connectivity by signed proximity estimation and pruning with deep nets. arXiv preprint arXiv:1807.02739 (2018)
  35. 35.
    Parag, T., Ciresan, D.C., Giusti, A.: Efficient classifier training to minimize false merges in electron microscopy segmentation. In: ICCV (2015)Google Scholar
  36. 36.
    Perez, A.J., et al.: A workflow for the automatic segmentation of organelles in electron microscopy image stacks. Front. Neuroanat. 8, 126 (2014)CrossRefGoogle Scholar
  37. 37.
    Plaza, S.M., Parag, T., Huang, G.B., Olbris, D.J., Saunders, M.A., Rivlin, P.K.: Annotating synapses in large EM datasets. arXiv preprint arXiv:1409.1801 (2014)
  38. 38.
    Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. In: Advances in Neural Information Processing Systems, pp. 91–99 (2015)Google Scholar
  39. 39.
    Roncal, W.G., et al.: VESICLE: volumetric evaluation of synaptic interfaces using computer vision at large scale. arXiv preprint arXiv:1403.3724 (2014)
  40. 40.
    Ronneberger, O., Fischer, P., Brox, T.: U-net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). Scholar
  41. 41.
    Roy, S., Namboodiri, V.P., Biswas, A.: Active learning with version spaces for object detection. arXiv preprint arXiv:1611.07285 (2016)
  42. 42.
    Scheffer, T., Decomain, C., Wrobel, S.: Active hidden Markov models for information extraction. In: Hoffmann, F., Hand, D.J., Adams, N., Fisher, D., Guimaraes, G. (eds.) IDA 2001. LNCS, vol. 2189, pp. 309–318. Springer, Heidelberg (2001). Scholar
  43. 43.
    Sener, O., Savarese, S.: Active learning for convolutional neural networks: a core-set approach. In: ICLR (2018)Google Scholar
  44. 44.
    Settles, B.: Active learning literature survey. Technical report, University of Wisconsin-Madison Department of Computer Sciences (2009)Google Scholar
  45. 45.
    Settles, B.: Active learning literature survey. 2010. Computer Sciences Technical Report (2014)Google Scholar
  46. 46.
    Seyedhosseini, M., Ellisman, M.H., Tasdizen, T.: Segmentation of mitochondria in electron microscopy images using algebraic curves. In: ISBI, pp. 860–863. IEEE (2013)Google Scholar
  47. 47.
    Simonyan, K., Zisserman, A.: Two-stream convolutional networks for action recognition in videos. In: NIPS (2014)Google Scholar
  48. 48.
    Sivaraman, S., Trivedi, M.M.: Active learning for on-road vehicle detection: a comparative study. Mach. Vis. Appl. 25, 599–611 (2014)CrossRefGoogle Scholar
  49. 49.
    Staffler, B., Berning, M., Boergens, K.M., Gour, A., van der Smagt, P., Helmstaedter, M.: SynEM, automated synapse detection for connectomics. Elife (2017)Google Scholar
  50. 50.
    Vazquez-Reina, A., Gelbart, M., Huang, D., Lichtman, J., Miller, E., Pfister, H.: Segmentation fusion for connectomics. In: ICCV (2011)Google Scholar
  51. 51.
    Vijayanarasimhan, S., Grauman, K.: Large-scale live active learning: training object detectors with crawled data and crowds. IJCV 108, 97–114 (2014)MathSciNetCrossRefGoogle Scholar
  52. 52.
    Wang, K., Zhang, D., Li, Y., Zhang, R., Lin, L.: Cost-effective active learning for deep image classification. IEEE TCSVT 27, 2591–2600 (2017)Google Scholar
  53. 53.
    Yoo, D., Kweon, I.S.: Learning loss for active learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 93–102 (2019)Google Scholar
  54. 54.
    Yu, K., Bi, J., Tresp, V.: Active learning via transductive experimental design. In: Proceedings of the 23rd International Conference on Machine Learning, pp. 1081–1088 (2006)Google Scholar
  55. 55.
    Zhang, Y., Lease, M., Wallace, B.C.: Active discriminative text representation learning. In: AAAI (2017)Google Scholar
  56. 56.
    Zheng, Z., et al.: A complete electron microscopy volume of the brain of adult drosophila melanogaster. Cell 174, 730–743 (2018)CrossRefGoogle Scholar
  57. 57.
    Zhu, X., Ghahramani, Z., Lafferty, J.D.: Semi-supervised learning using Gaussian fields and harmonic functions. In: ICML (2003)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Harvard UniversityCambridgeUSA
  2. 2.New York UniversityNew YorkUSA
  3. 3.MITCambridgeUSA
  4. 4.GoogleMountain ViewUSA
  5. 5.University of Massachusetts BostonBostonUSA
  6. 6.Broad InstituteCambridgeUSA
  7. 7.Comcast ResearchCambridgeUSA

Personalised recommendations