Skip to main content

Self-supervised Dense Representation Learning for Live-Cell Microscopy with Time Arrow Prediction

  • Conference paper
  • First Online:
Medical Image Computing and Computer Assisted Intervention – MICCAI 2023 (MICCAI 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14227))

  • 2657 Accesses

Abstract

State-of-the-art object detection and segmentation methods for microscopy images rely on supervised machine learning, which requires laborious manual annotation of training data. Here we present a self-supervised method based on time arrow prediction pre-training that learns dense image representations from raw, unlabeled live-cell microscopy videos. Our method builds upon the task of predicting the correct order of time-flipped image regions via a single-image feature extractor followed by a time arrow prediction head that operates on the fused features. We show that the resulting dense representations capture inherently time-asymmetric biological processes such as cell divisions on a pixel-level. We furthermore demonstrate the utility of these representations on several live-cell microscopy datasets for detection and segmentation of dividing cells, as well as for cell state classification. Our method outperforms supervised methods, particularly when only limited ground truth annotations are available as is commonly the case in practice. We provide code at https://github.com/weigertlab/tarrow.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 79.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: ICML, pp. 1597–1607 (2020)

    Google Scholar 

  2. Dorkenwald, M., Xiao, F., Brattoli, B., Tighe, J., Modolo, D.: SCVRL: shuffled contrastive video representation learning. In: CVPR, pp. 4132–4141 (2022)

    Google Scholar 

  3. Ericsson, L., Gouk, H., Loy, C.C., Hospedales, T.M.: Self-supervised representation learning: introduction, advances, and challenges. IEEE Sig. Process. Mag. 39(3), 42–62 (2022)

    Article  Google Scholar 

  4. Etournay, R., Popović, M., Merkel, M., Nandi, A., Blasse, C., Aigouy, B., et al.: Interplay of cell dynamics and epithelial tension during morphogenesis of the Drosophila pupal wing. eLife 4, e07090 (2015)

    Google Scholar 

  5. Gidaris, S., Singh, P., Komodakis, N.: Unsupervised representation learning by predicting image rotations. In: ICLR, OpenReview.net (2018)

    Google Scholar 

  6. Greenwald, N.F., Miller, G., Moen, E., Kong, A., Kagel, A., et al.: Whole-cell segmentation of tissue images with human-level performance using large-scale data annotation and deep learning. Nat. Biotechnol. 40(4), 555–565 (2021)

    Article  Google Scholar 

  7. Han, H., Dmitrieva, M., Sauer, A., Tam, K.H., Rittscher, J.: Self-supervised voxel-level representation rediscovers subcellular structures in volume electron microscopy. In: CVPRW, pp. 1874–1883 (2022)

    Google Scholar 

  8. He, K., Chen, X., Xie, S., Li, Y., Dollár, P., Girshick, R.: Masked autoencoders are scalable vision learners. In: CVPR, pp. 16000–16009 (2022)

    Google Scholar 

  9. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR, pp. 770–778 (2016)

    Google Scholar 

  10. Hsu, J., Gu, J., Wu, G., Chiu, W., Yeung, S.: Capturing implicit hierarchical structure in 3d biomedical images with self-supervised hyperbolic representations. In: NeurIPS, vol. 34, pp. 5112–5123 (2021)

    Google Scholar 

  11. Hu, K., Shao, J., Liu, Y., Raj, B., Savvides, M., Shen, Z.: Contrast and order representations for video self-supervised learning. In: ICCV, pp. 7939–7949 (2021)

    Google Scholar 

  12. Hua, T., Wang, W., Xue, Z., Ren, S., Wang, Y., Zhao, H.: On feature decorrelation in self-supervised learning. In: CVPR, pp. 9598–9608 (2021)

    Google Scholar 

  13. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: ICLR (2015)

    Google Scholar 

  14. Lee, H.Y., Huang, J.B., Singh, M., Yang, M.H.: Unsupervised representation learning by sorting sequences. In: ICCV, pp. 667–676 (2017)

    Google Scholar 

  15. Misra, I., Zitnick, C.L., Hebert, M.: Shuffle and learn: unsupervised learning using temporal order verification. In: ECCV, pp. 527–544 (2016)

    Google Scholar 

  16. Padovani, F., Mairhörmann, B., Falter-Braun, P., Lengefeld, J., Schmoller, K.M.: Segmentation, tracking and cell cycle analysis of live-cell imaging data with Cell-ACDC. BMC Biol. 20, 174 (2022)

    Article  Google Scholar 

  17. Padovani, F., Mairhörmann, B., Lengefeld, J., Falter-Braun, P., Schmoller, K.: Cell-ACDC: segmentation, tracking, annotation and quantification of microscopy imaging data (dataset). https://zenodo.org/record/6795124 (2022)

  18. Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., Efros, A.A.: Context encoders: feature learning by inpainting. In: CVPR, pp. 2536–2544 (2016)

    Google Scholar 

  19. Pickup, L.C., et al.: Seeing the arrow of time. In: CVPR, pp. 2043–2050 (2014)

    Google Scholar 

  20. Piscitello-Gómez, R., Gruber, F.S., Krishna, A., Duclut, C., Modes, C.D., et al.: Core PCP mutations affect short time mechanical properties but not tissue morphogenesis in the Drosophila pupal wing. bioRxiv (2022)

    Google Scholar 

  21. Ronneberger, O., Fischer, P., Brox, T.: U-net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28

    Chapter  Google Scholar 

  22. Schiappa, M.C., Rawat, Y.S., Shah, M.: Self-supervised learning for videos: a survey. ACM Comput. Surv. 55(13s), 1–37 (2022)

    Article  Google Scholar 

  23. Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-CAM: visual explanations from deep networks via gradient-based localization. In: ICCV, pp. 618–626 (2017)

    Google Scholar 

  24. Stelzer, E.H.K., et al.: Light sheet fluorescence microscopy. Nat. Rev. Methods Primers 1(1), 1–25 (2021)

    Article  MathSciNet  Google Scholar 

  25. Stringer, C., Wang, T., Michaelos, M., Pachitariu, M.: Cellpose: a generalist algorithm for cellular segmentation. Nat. Methods 18(1), 100–106 (2021)

    Article  Google Scholar 

  26. Tomer, R., Khairy, K., Keller, P.J.: Shedding light on the system: studying embryonic development with light sheet microscopy. Curr. Opin. Genet. Dev. 21(5), 558–565 (2011)

    Article  Google Scholar 

  27. Ulicna, K., Vallardi, G., Charras, G., Lowe, A.: Mdck cell tracking reference dataset. https://rdr.ucl.ac.uk/articles/dataset/Cell_tracking_reference_dataset/16595978

  28. Ulicna, K., Vallardi, G., Charras, G., Lowe, A.R.: Automated deep lineage tree analysis using a bayesian single cell tracking approach. Front. Comput. Sci. 3, 734559 (2021)

    Article  Google Scholar 

  29. Ulman, V., Maška, M., Magnusson, K.E.G., Ronneberger, O., Haubold, C., et al.: An objective comparison of cell-tracking algorithms. Nat. Methods 14(12), 1141–1152 (2017). https://doi.org/10.1038/nmeth.4473

    Article  Google Scholar 

  30. Wei, D., Lim, J., Zisserman, A., Freeman, W.T.: Learning and using the arrow of time. In: CVPR, pp. 8052–8060 (2018)

    Google Scholar 

  31. Weigert, M., Schmidt, U., Haase, R., Sugawara, K., Myers, G.: Star-convex polyhedra for 3d object detection and segmentation in microscopy. In: WACV, pp. 3666–3673 (2020)

    Google Scholar 

  32. Zaheer, M., Kottur, S., Ravanbakhsh, S., Poczos, B., Salakhutdinov, R.R., Smola, A.J.: Deep sets. In: NeurIPS (2017)

    Google Scholar 

  33. Zbontar, J., Jing, L., Misra, I., LeCun, Y., Deny, S.: Barlow twins: self-supervised learning via redundancy reduction. In: ICML, pp. 12310–12320 (2021)

    Google Scholar 

Download references

Acknowledgements

We thank Albert Dominguez (EPFL) and Uwe Schmidt for helpful comments, Natalie Dye (PoL Dresden) and Franz Gruber for providing the Flywing dataset, Benedikt Mairhörmann and Kurt Schmoller (Helmholtz Munich) for providing additional Yeast training data, and Alan Lowe (UCL) for providing the Mdck dataset. M.W. and B.G. are supported by the EPFL School of Life Sciences ELISIR program and CARIGEST SA.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Martin Weigert .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Gallusser, B., Stieber, M., Weigert, M. (2023). Self-supervised Dense Representation Learning for Live-Cell Microscopy with Time Arrow Prediction. In: Greenspan, H., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2023. MICCAI 2023. Lecture Notes in Computer Science, vol 14227. Springer, Cham. https://doi.org/10.1007/978-3-031-43993-3_52

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-43993-3_52

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-43992-6

  • Online ISBN: 978-3-031-43993-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics