Abstract
Cell tracking in microscopy images is fundamental to new biological and medical discoveries today. It facilitates the study of the properties of living cells over time. Due to the temporal nature of the cell tracking task, data association is the most difficult aspect of automated cell tracking. Due to the intricate nature of biological, imaging, and algorithmic factors that influence cell segmentation and tracking results, it is challenging to provide a simple and efficient method to determine the appropriate approach for a specific dataset. For example, mitosis, a crucial biological process, plays a key role in correcting trajectories. This research looked at all the challenges of the cell tracking task and the current solutions that have been proposed so far. In this paper, we carefully identified the sources of the tracking challenges and categorized them in a hierarchical diagram to explain the impact of challenges at different levels on the different cell tracking subtasks. Then, after identifying the solutions provided so far, we classified them into three levels: strategic, tactical, and technical. At the strategy level, tracking before detection and tracking by detection are two main approaches. The tactics can be based on cell distance, similarity, overlap, motions, probability, model evaluation, and deep-learning methods. The techniques identified in our analysis include contour evolution, nearest-neighbor linking, morphological-operator-based tracking, similarity-based label propagation, overlap-based label propagation, motion-prediction-based label propagation, graph-based multiple hypothesis tracking, probability-graph-based global optimization, probability-model-based global optimization, and recently developed deep-learning models. By merging cell tracking methods at different levels in one diagram, this classification will help to understand current solutions and provide new insights for cell tracking algorithms. Overall, in this study, we conducted a comprehensive investigation of the challenges of cell tracking and its corresponding solutions, offering a unique source of information.
Similar content being viewed by others
Data Availability
No new data were created or analyzed in this study
References
Mavska M, Ulman V, Svoboda D, Matula P, Matula P, Ederra C, Urbiola A, Espana T, Venkatesan S, Balak DMW, Karas P, Bolckova T, Streitova M, Carthel CA, Coraluppi SP, Harder N, Rohr K, Magnusson KEG, Jalden J, Blau HM, Dzyubachyk O, Krizek P, Hagen GM, Pastor-Escuredo D, Jimenez-Carretero D, Ledesma-Carbayo MJ, Munoz-Barrutia A, Meijering EHW, Kozubek M, Ortiz-de-Solorzano C (2014) A benchmark for comparison of cell tracking algorithms. Bioinformatics 30(11):1609–1617. https://doi.org/10.1093/bioinformatics/btu080
Ong JY, Torres JZ (2019) Dissecting the mechanisms of cell division. J Biol Chem 294:11382–11390. https://doi.org/10.1074/jbc.AW119.008149
Li X, Miao Y, Pal D, Devreotes P (2020) Excitable networks controlling cell migration during development and disease. Semin Cell Dev Biol 100:133–142. https://doi.org/10.1016/j.semcdb.2019.11.001
Freitas JT, Jozic I, Bedogni B (2021) Wound healing assay for melanoma cell migration. Methods Mol 2265:65–71
Liu JC, Zacksenhouse M, Eisen A, Nofech-Mozes S, Zacksenhaus E (2017) Identification of cell proliferation, immune response and cell migration as critical pathways in a prognostic signature for her2+:er\(\alpha \)- breast cancer. PLoS ONE 12(6):0179223. https://doi.org/10.1371/journal.pone.0179223
Anjum S, Gurari D (2020) Ctmc: Cell tracking with mitosis detection dataset challenge. In: 2020 IEEE/CVF conference on computer vision and pattern recognition workshops (CVPRW), pp 4228–4237. https://doi.org/10.1109/CVPRW50498.2020.00499
Amat F, Lemon WC, Mossing DP, McDole K, Wan Y, Branson K, Myers EW, Keller PJ (2014) Fast, accurate reconstruction of cell lineages from large-scale fluorescence microscopy data. Nat Methods 11:951–958
Chen X, Zhou X, Wong STC (2006) Automated segmentation, classification, and tracking of cancer cell nuclei in time-lapse microscopy. IEEE Trans Biomed Eng 53:762–766
Yang X, Li H, Zhou X (2006) Nuclei segmentation using marker-controlled watershed, tracking using mean-shift, and kalman filter in time-lapse microscopy. IEEE Trans Circuits Syst I: Regul Pap 53:2405–2414
Li F, Zhou X, Ma J, Wong STC (2010) Multiple nuclei tracking using integer programming for quantitative cancer cell cycle analysis. IEEE Trans Med Imaging 29:96–105. https://doi.org/10.1109/TCSI.2006.884469
Holme B, Bjørnerud B, Pedersen NM, Ballina LR, Wesche J, Haugsten EM (2023) Automated tracking of cell migration in phase contrast images with celltraxx. Sci Rep 13. https://doi.org/10.1038/s41598-023-50227-9
Dzyubachyk O, Cappellen WA, Essers J, Niessen WJ, Meijering EHW (2010) Advanced level-set-based cell tracking in time-lapse fluorescence microscopy. IEEE Trans Med Imaging 29:852–867
Svoboda D, Ulman V (2017) Mitogen: A framework for generating 3d synthetic time-lapse sequences of cell populations in fluorescence microscopy. IEEE Trans Med Imaging 36:310–321
Kok RNU, Hebert L, Huelsz-Prince G, Goos YJ, Zheng X, Bozek K, Stephens GJ, Tans SJ, Zon JS (2020) Organoidtracker: Efficient cell tracking using machine learning and manual error correction. PLoS ONE 15(10):0240802. https://doi.org/10.1371/journal.pone.0240802
Magnusson KEG (2016) Segmentation and tracking of cells and particles in time-lapse microscopy. PhD thesis, Royal Institute of Technology, Stockholm, Sweden https://nbn-resolving.org/urn:nbn:se:kth:diva-196911
Maddalena L, Antonelli L, Albu A-I, Hada A, Guarracino MR (2022) Artificial intelligence for cell segmentation, event detection, and tracking for label-free microscopy imaging. Algorithms 15:313
...Ulman V, Maska M, Magnusson KEG, Ronneberger O, Haubold C, Harder N, Matula P, Matula P, Svoboda D, Radojevic M, Smal I, Rohr K, Jalden J, Blau HM, Dzyubachyk O, Lelieveldt BPF, Xiao P, Li Y, Cho S-Y, Dufour AC, Olivo-Marin J-C, Reyes-Aldasoro CC, Solis-Lemus JA, Bensch R, Brox T, Stegmaier J, Mikut R, Wolf S, Hamprecht FA, Esteves T, Quelhas P, Demirel OB, Malmstrom L, Jug F, Tomançak P, Meijering EHW, Munoz-Barrutia A, Kozubek M, Ortiz-de-Solorzano C (2017) An objective comparison of cell tracking algorithms. Nat Methods 14:1141–1152
...Ker DFE, Eom S, Sanami S, Bise R, Pascale C, Yin Z, Huh S, Osuna-Highley E, Junkers S, Helfrich CJ, Liang PY, Pan J, Jeong S, Kang SS, Liu J, Nicholson R, Sandbothe MF, Van PT, Liu A, Chen M, Kanade T, Weiss LE, Campbell PG (2018) Phase contrast time-lapse microscopy datasets with automated and manual cell tracking annotations. Sci Data 5:180237. https://doi.org/10.1038/sdata.2018.237
Matula P, Maska M, Sorokin DV, Matula P, Ortiz-de-Solorzano C, Kozubek M (2015) Cell tracking accuracy measurement based on comparison of acyclic oriented graphs. PLoS ONE 10(12):0144959. https://doi.org/10.1371/journal.pone.0144959
Meijering E (2012) Cell segmentation: 50 years down the road [life sciences]. IEEE Signal Process Mag 29(5):140–145. https://doi.org/10.1109/MSP.2012.2204190
Maska M, Ulman V, Delgado-Rodriguez P, Gomez-de-Mariscal E, Necasova T, Guerrero Pena FA, Ren TI, Meyerowitz EM, Scherr T, Loffler K, Mikut R, Guo T, Wang Y, Allebach JP, Bao R, Al-Shakarji NM, Rahmon G, Toubal IE, Palaniappan K, Lux F, Matula P, Sugawara K, Magnusson KEG, Aho L, Cohen AR, Arbelle A, Ben-Haim T, Raviv TR, Isensee F, Jager PF, Maier-Hein KH, Zhu Y, Ederra C, Urbiola A, Meijering E, Cunha A, Munoz-Barrutia A, Kozubek M, Ortiz-de-Solorzano C (2023) The cell tracking challenge: 10 years of objective benchmarking. Nat Methods 20:1010–1020. https://doi.org/10.1038/s41592-023-01879-y
Lux F, Matula P (2019) Dic image segmentation of dense cell populations by combining deep learning and watershed. In: 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019), pp 236–239. 10.1109/ISBI.2019.8759594
Ren W, Wang X, Tian J, Tang Y, Chan AB (2021) Tracking-by-counting: Using network flows on crowd density maps for tracking multiple targets. IEEE Trans Image Process 30:1439–1452
Wang Z, Yin L, Wang Z (2019) A new approach for cell detection and tracking. IEEE Access 7:99889–99899. https://doi.org/10.1109/ACCESS.2019.2930539
Scherr T, Löffler K, Böhland M, Mikut R (2020) Cell segmentation and tracking using cnn-based distance predictions and a graph-based matching strategy. PLoS ONE 15(12):0243219. https://doi.org/10.1371/journal.pone.0243219
Hayashida J, Bise R (2019) Cell tracking with deep learning for cell detection and motion estimation in low-frame-rate. In: International conference on medical image computing and computer-assisted intervention, vol 11764, pp 397–405. https://doi.org/10.1007/978-3-030-32239-7_44
Bao R, Al-Shakarji NM, Bunyak F, Palaniappan K (2021) Dmnet: Dual-stream marker guided deep network for dense cell segmentation and lineage tracking. In: 2021 IEEE/CVF international conference on computer vision workshops (ICCVW), pp 3354–3363. https://doi.org/10.1109/ICCVW54120.2021.00375
Akbas CE, Ulman V, Maska M, Jug F, Kozubek M (2019) Automatic fusion of segmentation and tracking labels. In: ECCV 2018, vol 11134, pp 446–454. https://doi.org/10.1007/978-3-030-11024-6_34
Rahmon, G., Bunyak, F., Seetharaman, G., Palaniappan, K.: Motion u-net: Multi-cue encoder-decoder network for motion segmentation. In: 2020 25th International conference on pattern recognition (ICPR), pp. 8125–8132 (2021). https://doi.org/10.1109/ICPR48806.2021.9413211
Zhu Y, Meijering EHW (2021) Automatic improvement of deep learning-based cell segmentation in time-lapse microscopy by neural architecture search. Bioinformatics 37:4844–4850
Isensee F, Jaeger PF, Kohl SAA, Petersen J, Maier-Hein K (2020) nnu-net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods 18:203–211. https://doi.org/10.1038/s41592-020-01008-z
Eschweiler D, Spina TV, Choudhury RC, Meyerowitz E, Cunha A, Stegmaier J (2019) Cnn-based preprocessing to optimize watershed-based cell segmentation in 3d confocal microscopy images. In: 16th IEEE International Symposium on Biomedical Imaging (ISBI), pp 223–227. https://doi.org/10.1109/ISBI.2019.8759242
Ronneberger O, Fischer P, Brox T (2015) U-net: Convolutional networks for biomedical image segmentation. In: Medical Image Computing and Computer-Assisted Intervention - MICCAI 2015, vol 9351, pp 234–241. https://doi.org/10.1007/978-3-319-24574-4_28
Zhou Z, Siddiquee MMR, Tajbakhsh N, Liang J (2018) Unet++: A nested u-net architecture for medical image segmentation. In: Deep Learning in Medical Image Analysis - and - Multimodal Learning for Clinical Decision Support - 4th International Workshop, DLMIA 2018, and 8th International Workshop, ML-CDS 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, September 20, 2018, Proceedings, vol 11045, pp 3–11. https://doi.org/10.1007/978-3-030-00889-5_1
Françani, AO (2022) Analysis of the performance of U-Net neural networks for the segmentation of living cells
Kumar M, Mondal S (2021) Recent developments on target tracking problems: A review. Ocean Engineering 236:109558. https://doi.org/10.1016/j.oceaneng.2021.109558
Dendorfer P, Osep A, Milan A, Schindler K, Cremers D, Reid ID, Roth S, Leal-Taixé L (2021) Motchallenge: A benchmark for single-camera multiple target tracking. Int J Comput Vis 129:845–881. https://doi.org/10.1007/s11263-020-01393-0
Yao Y, Smal I, Grigoriev I, Akhmanova A, Meijering EHW (2020) Deep-learning method for data association in particle tracking. Bioinformatics 36(16):4935–4941. https://doi.org/10.1093/bioinformatics/btaa597
Funamoto K, Zervantonakis IK, Liu Y, Ochs CJ, Kim C, Kamm RD (2012) A novel microfluidic platform for high-resolution imaging of a three-dimensional cell culture under a controlled hypoxic environment. Lab on a Chip 12(22):4855–63
Ananthakrishnan R, Ehrlicher AJ (2007) The forces behind cell movement. Int J Biol Sci 3:303–317
Ortiz-de-Solórzano C, Muñoz-Barrutia A, Meijering EHW, Kozubek M (2015) Toward a morphodynamic model of the cell: Signal processing for cell modeling. IEEE Signal Process Mag 32:20–29
Gabillon Y, Lepreux S, Oliveira KM (2013) Towards ergonomic user interface composition: A study about information density criterion. In: Human-Computer Interaction. Human-Centred Design Approaches, Methods, Tools, and Environments, pp 211–220. Springer, Berlin, Heidelberg
Jahn SW, Plass M, Moinfar F (2020) Digital pathology: Advantages, limitations and emerging perspectives. J Clin Med 9(11):3697. https://doi.org/10.3390/jcm9113697
Wikipedia contributors (2021) Soft ergonomics — Wikipedia, The Free Encyclopedia. Online; Accessed 22-Jan-2022. https://en.wikipedia.org/w/index.php?title=Soft_ergonomics &oldid=1029247135
Lu Y, Liu A-A, Su Y-T (2021) Chapter 6 - mitosis detection in biomedical images. In: Chen, M. (ed.) Computer Vision for Microscopy Image Analysis. Computer Vision and Pattern Recognition, pp 131–157. https://doi.org/10.1016/B978-0-12-814972-0.00006-0
Gilad T, Reyes J, Chen J-Y, Lahav G, Riklin-Raviv T (2019) Fully unsupervised symmetry-based mitosis detection in time-lapse cell microscopy. Bioinformatics 35(15):2644–2653. https://doi.org/10.1093/bioinformatics/bty1034
Su Y-T, Lu Y, Chen M, Liu A-A (2022) Deep reinforcement learning-based progressive sequence saliency discovery network for mitosis detection in time-lapse phase-contrast microscopy images. IEEE/ACM Trans Comput Biol Bioinform 19(2):854–865. https://doi.org/10.1109/TCBB.2020.3019042
Chen Y, Huo Y (2020) Limitation of Acyclic Oriented Graphs Matching as Cell Tracking Accuracy Measure when Evaluating Mitosis
Winter MR, Mankowski WC, Wait E, Hoz EC, Aguinaldo A, Cohen AR (2019) Separating touching cells using pixel replicated elliptical shape models. IEEE Trans Med Imaging 38:883–893
Guerrero Peña FA, Marrero Fernandez PD, Tarr PT, Ren TI, Meyerowitz EM, Cunha A (2020) J regularization improves imbalanced multiclass segmentation. In: 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI), pp 1–5. https://doi.org/10.1109/ISBI45749.2020.9098550
Scherr T, Löffler K, Neumann O, Mikut R (2021) On improving an already competitive segmentation algorithm for the cell tracking challenge - lessons learned. bioRxiv
Arbelle A, Raviv TR (2019) Microscopy cell segmentation via convolutional lstm networks. In: 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019), pp 1008–1012. https://doi.org/10.1109/ISBI.2019.8759447
Chen M (2021) Chapter 5 - cell tracking in time-lapse microscopy image sequences. In: Chen, M. (ed.) Computer Vision for Microscopy Image Analysis. Computer Vision and Pattern Recognition, pp 101–129. https://doi.org/10.1016/B978-0-12-814972-0.00005-9
Spilger R, Imle A, Lee J-Y, Müller B, Fackler OT, Bartenschlager R, Rohr K (2020) A recurrent neural network for particle tracking in microscopy images using future information, track hypotheses, and multiple detections. IEEE Trans Image Process 29:3681–3694. https://doi.org/10.1109/TIP.2020.2964515
Sixta T, Cao J, Seebach J, Schnittler H, Flach B (2020) Coupling cell detection and tracking by temporal feedback. Machine Vision and Applications 31(24):1. https://doi.org/10.1007/s00138-020-01072-7
Huang, L., McKay, G.N., Durr, N.: A deep learning bidirectional temporal tracking algorithm for automated blood cell counting from non-invasive capillaroscopy videos. In: MICCAI 2021, vol 12908 (2021). https://doi.org/10.1007/978-3-030-87237-3_40
Gilad T, Bray M-A, Carpenter AE, Raviv TR (2015) Symmetry-based mitosis detection in time-lapse microscopy. In: 2015 IEEE 12th International Symposium on Biomedical Imaging (ISBI), pp 164–167. https://doi.org/10.1109/ISBI.2015.7163841
Skylaki S, Hilsenbeck O, Schroeder T (2016) Challenges in long-term imaging and quantification of single-cell dynamics. Nat Biotechnol 34:1137–1144
Wikipedia contributors (2020) Batch effect — Wikipedia, The Free Encyclopedia. Online; Accessed 22-Jan-2022. https://en.wikipedia.org/w/index.php?title=Batch_effect &oldid=991656467
Leek JT, Scharpf RB, Bravo HC, Simcha DM, Langmead B, Johnson W, Geman D, Baggerly KA, Irizarry RA (2010) Tackling the widespread and critical impact of batch effects in high-throughput data. Nat Rev Genet 11:733–739
Johnson W, Li C, Rabinovic A (2007) Adjusting batch effects in microarray expression data using empirical bayes methods. Biostatistics (Oxford, England) 8:118–27. https://doi.org/10.1093/biostatistics/kxj037
Ramesh N, Tasdizen T (2021) Chapter 3 - detection and segmentation in microscopy images. In: Chen, M. (ed.) Computer Vision for Microscopy Image Analysis. Computer Vision and Pattern Recognition, pp 43–71. https://doi.org/10.1016/B978-0-12-814972-0.00003-5
Panteli A, Gupta DK, Bruijn N, Gavves E (2020) Siamese tracking of cell behaviour patterns. In: Proceedings of the Third Conference on Medical Imaging with Deep Learning. Proceedings of Machine Learning Research, vol 121, pp 570–587
Harder N, Mora-Bermúdez F, Godinez WJ, Wünsche A, Eils R, Ellenberg J, Rohr K (2009) Automatic analysis of dividing cells in live cell movies to detect mitotic delays and correlate phenotypes in time. Genome Res 19(11):2113–24
Wollmann T, Gunkel M, Chung I, Erfle H, Rippe K, Rohr K (2019) Gruu-net: Integrated convolutional and gated recurrent neural network for cell segmentation. Med Image Anal 56:68–79. https://doi.org/10.1016/j.media.2019.04.011
Shi J, Xu B, Zhu P, Lu M (2016) Multi-task firework algorithm for cell tracking and contour estimation. In: 2016 International Conference on Control, Automation and Information Sciences (ICCAIS), pp 27–31. https://doi.org/10.1109/ICCAIS.2016.7822430
Forero MG, Morales KD (2021) Evaluation of filtering techniques for cell tracking in confocal microscopy images. In: Tescher AG, Ebrahimi T (eds) Applications of Digital Image Processing XLIV, vol 11842, p 1184214. https://doi.org/10.1117/12.2594392. International Society for Optics and Photonics
Taghanaki SA, Zheng Y, Zhou SK, Georgescu B, Sharma PS, Xu D, Comaniciu D, Hamarneh G (2019) Combo loss: Handling input and output imbalance in multi-organ segmentation. Comput Medical Imaging Graph 75:24–33. https://doi.org/10.1016/j.compmedimag.2019.04.005
Wikipedia contributors: MeVisLab — Wikipedia, The Free Encyclopedia. Online; Accessed 25-Jan-2022 (2021). https://en.wikipedia.org/w/index.php?title=MeVisLab &oldid=1043369447
Schindelin JE, Arganda-Carreras I, Frise E, Kaynig V, Longair M, Pietzsch T, Preibisch S, Rueden CT, Saalfeld S, Schmid B, Tinevez J-Y, White DJ, Hartenstein V, Eliceiri KW, Tomançak P, Cardona A (2012) Fiji: an open-source platform for biological-image analysis. Nat Methods 9:676–682. https://doi.org/10.1038/nmeth.2019
Chen M (2021) Chapter 9 - open data and software for microscopy image analysis. In: Chen, M. (ed.) Computer Vision for Microscopy Image Analysis. Computer Vision and Pattern Recognition, pp 203–208. https://doi.org/10.1016/B978-0-12-814972-0.00009-6
Shankar K, Perumal E, Elhoseny M, Taher F, Gupta BB, El-Latif AAA (2021) Synergic deep learning for smart health diagnosis of Covid-19 for connected living and smart cities. ACM Trans Internet Technol 22(3):1. https://doi.org/10.1145/3453168
Winter MR, Mankowski WC, Wait E, Temple S, Cohen AR (2016) Lever: software tools for segmentation, tracking and lineaging of proliferating cells. Bioinformatics 32(22):3530–3531
Cordelières FP, Petit V, Kumasaka MY, Debeir O, Letort V, Gallagher SJ, Larue L (2013) Automated cell tracking and analysis in phase-contrast videos (itrack4u): Development of java software based on combined mean-shift processes. PLoS ONE 8(11):81266. https://doi.org/10.1371/journal.pone.0081266
Aragaki H, Ogoh K, Kondo Y, Aoki K (2022) Lim tracker: a software package for cell tracking and analysis with advanced interactivity. Sci Rep 12:2702
Nishimura, K., Bise, R.: Spatial-temporal mitosis detection in phase-contrast microscopy via likelihood map estimation by 3dcnn. In: 2020 42nd Annual international conference of the IEEE engineering in medicine and biology society (EMBC), pp. 1811–1815 (2020). https://doi.org/10.1109/EMBC44109.2020.9175676
Ciresan DC, Giusti A, Gambardella LM, Schmidhuber J (2013) Mitosis detection in breast cancer histology images with deep neural networks. In: Medical image computing and computer-assisted intervention (MICCAI), vol 16, pp 411–418
Sebai M, Wang X, Wang T (2020) Maskmitosis: a deep learning framework for fully supervised, weakly supervised, and unsupervised mitosis detection in histopathology images. Med Biol Eng Comput 58(7):1603–1623. https://doi.org/10.1007/s11517-020-02175-z
Mao Y, Yin Z (2016) A hierarchical convolutional neural network for mitosis detection in phase-contrast microscopy images, vol 9901, pp 685–692. https://doi.org/10.1007/978-3-319-46723-8_79
Chen L-C, Zhu Y, Papandreou G, Schroff F, Adam H (2018) Encoder-decoder with atrous separable convolution for semantic image segmentation. In: European Conference on Computer Vision–ECCV 2018, vol 11211, pp 833–851. https://doi.org/10.1007/978-3-030-01234-2_49
Mahmood T, Arsalan M, Owais M, Lee MB, Park KR (2020) Artificial intelligence-based mitosis detection in breast cancer histopathology images using faster r-cnn and deep cnns. J Clin Med 9(3):749. https://doi.org/10.3390/jcm9030749
Li Y, Rose F, Pietro F, Morin X, Genovesio A (2016) Detection and tracking of overlapping cell nuclei for large scale mitosis analyses. BMC Bioinform 17:183. https://doi.org/10.1186/s12859-016-1030-9
Harder N, Mora-Bermudez F, Godinez WJ, Ellenberg J, Eils R, Rohr K (2007) Determination of mitotic delays in 3d fluorescence microscopy images of human cells using an error-correcting finite state machine. In: 2007 4th IEEE International Symposium on Biomedical Imaging: From Nano to Macro, pp 1044–1047. https://doi.org/10.1109/ISBI.2007.357034
Mao Y, Han L, Yin Z (2019) Cell mitosis event analysis in phase contrast microscopy images using deep learning. Med Image Anal 57:32–43
Wu B, Kausar T, Xiao Q, Wang M, Wang W, Fan B, Sun D (2017) Ff-cnn: An efficient deep neural network for mitosis detection in breast cancer histological images. In: Medical Image Understanding and Analysis, pp 249–260. https://doi.org/10.1007/978-3-319-60964-5_22
Gallardo GM, Yang F, Ianzini F, Mackey MA, Sonka M (2004) Mitotic cell recognition with hidden markov models. In: Medical Imaging 2004: Visualization, Image-Guided Procedures, and Display, San Diego, California, United States, 14-19 February 2004. SPIE Proceedings, vol 5367. https://doi.org/10.1117/12.535778
Liang L, Zhou X, Li F, Wong ST, Huckins J, King RW (2007) Mitosis cell identification with conditional random fields. In: 2007 IEEE/NIH Life Science Systems and Applications Workshop, pp 9–12. https://doi.org/10.1109/LSSA.2007.4400872
Liu A, Li K, Kanade T (2010) Mitosis sequence detection using hidden conditional random fields. In: 2010 IEEE International Symposium on Biomedical Imaging: From Nano to Macro, pp 580–583. https://doi.org/10.1109/ISBI.2010.5490279
Huh S, Chen M (2011) Detection of mitosis within a stem cell population of high cell confluence in phase-contrast microscopy images. CVPR 2011:1033–1040. https://doi.org/10.1109/CVPR.2011.5995717
Liu A, Li K, Kanade T (2012) A semi-markov model for mitosis segmentation in time-lapse phase contrast microscopy image sequences of stem cell populations. IEEE Trans Med Imaging 31:359–369
Lu Y, Liu A, Chen M, Nie W, Su Y (2020) Sequential saliency guided deep neural network for joint mitosis identification and localization in time-lapse phase contrast microscopy images. IEEE J Biomed Health Inf 24:1367–1378
Su Y, Lu Y, Chen M, Liu A (2017) Spatiotemporal joint mitosis detection using cnn-lstm network in time-lapse phase contrast microscopy images. IEEE Access 5:18033–18041
Zhou Y, Mao H, Yi Z (2017) Cell mitosis detection using deep neural networks. Knowl Based Syst 137:19–28
Nie W-Z, Li W-H, Liu A-A, Hao T, Su Y-T (2016) 3d convolutional networks-based mitotic event detection in time-lapse phase contrast microscopy image sequences of stem cell populations. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp 1359–1366. https://doi.org/10.1109/CVPRW.2016.171
Nie W, Yan Y, Hao T, Liu C, Su Y (2018) Mitosis event recognition and detection based on evolution of feature in time domain. Mach Vis Appl 29:1249–1256. https://doi.org/10.1007/s00138-018-0913-3
Su Y, Lu Y, Liu J, Chen M, Liu A (2021) Spatio-temporal mitosis detection in time-lapse phase-contrast microscopy image sequences: A benchmark. IEEE Trans Med Imaging 40:1319–1328
Ma M, Shi Y, Li W, Gao Y, Xu J (2018) A novel two-stage deep method for mitosis detection in breast cancer histology images. In: 2018 24th International Conference on Pattern Recognition (ICPR), pp 3892–3897. https://doi.org/10.1109/ICPR.2018.8546192
Jiang J, Khan A, Shailja S, Belteton SA, Goebel M, Szymanski DB, Manjunath BS (2023) Segmentation, tracking, and sub-cellular feature extraction in 3d time-lapse images. Sci Rep 13:1. https://doi.org/10.1038/s41598-023-29149-z
Xu J, Zhou D, Deng D, Li J, Chen C, Liao X, Chen G, Heng P-A (2022) Deep learning in cell image analysis. Intell Comput 3:1–15. https://doi.org/10.34133/2022/9861263
Maska M, Munoz-Barrutia A, Ortiz-de-Solórzano C (2012) Fast tracking of fluorescent cells based on the chan-vese model. In: 2012 9th IEEE International Symposium on Biomedical Imaging (ISBI), pp 1316–1319. https://doi.org/10.1109/ISBI.2012.6235805
Mavska M, Danek O, Garasa S, Rouzaut A, Munoz-Barrutia A, Ortiz-de-Solorzano C (2013) Segmentation and shape tracking of whole fluorescent cells based on the chan-vese model. IEEE Trans Med Imaging 32:995–1006
Jo H, Han J, Kim YS, Lee Y, Yang S (2021) A novel method for effective cell segmentation and tracking in phase contrast microscopic images. Sensors 21(10):3516. https://doi.org/10.3390/s21103516
Yu S, Lu Y, Molloy D (2019) A dynamic-shape-prior guided snake model with application in visually tracking dense cell populations. IEEE Trans Image Process 28(3):1513–1527. https://doi.org/10.1109/TIP.2018.2878331
Versari C, Stoma S, Batmanov K, Llamosi A, Mroz F, Kaczmarek A, Deyell M, Lhoussaine C, Hersen P, Batt G (2017) Long-term tracking of budding yeast cells in brightfield microscopy: Cellstar and the evaluation platform. J R Soc Interface 14(127):20160705
Dufour AC, Shinin V, Tajbakhsh S, Guillen-Aghion N, Olivo-Marin J-C, Zimmer C (2005) Segmenting and tracking fluorescent cells in dynamic 3-d microscopy with coupled active surfaces. IEEE Trans Image Process 14:1396–1410
Padfield DR, Rittscher J, Thomas N, Roysam B (2009) Spatio-temporal cell cycle phase analysis using level sets and fast marching methods. Med Image Anal 13(1):143–55
Zhao M, Jha A, Liu Q, Millis BA, Mahadevan-Jansen A, Lu L, Landman BA, Tyska MJ (2021) Faster mean-shift: Gpu-accelerated clustering for cosine embedding-based cell segmentation and tracking. Med Image Anal 71:102048. https://doi.org/10.1016/j.media.2021.102048
Debeir O, Ham PV, Kiss R, Decaestecker C (2005) Tracking of migrating cells under phase-contrast video microscopy with combined mean-shift processes. IEEE Trans Med Imaging 24:697–711
Castilla C, Maška M, Sorokin DV, Meijering E, Ortiz-de-Solórzano C (2019) 3-d quantification of filopodia in motile cancer cells. IEEE Trans Med Imaging 38(3):862–872. https://doi.org/10.1109/TMI.2018.2873842
Liang P, Chen J, Zhang Y, Wang H, Zheng H, Gu P, Chen D (2020) Intracker: An integrated detector-tracker framework for cell detection and tracking. In: 2020 IEEE 33rd International Symposium on Computer-Based Medical Systems (CBMS), pp 332–337. https://doi.org/10.1109/CBMS49503.2020.00069
Payer C, Stern D, Neff T, Bischof H, Urschler M (2018) Instance segmentation and tracking with cosine embeddings and recurrent hourglass networks. In: Medical Image Computing and Computer Assisted Intervention - MICCAI 2018 - 21st International Conference, Granada, Spain, September 16-20, 2018, Proceedings, Part II. Lecture Notes in Computer Science, vol 11071, pp 3–11. https://doi.org/10.1007/978-3-030-00934-2_1
Shailja S, Jiang J, Manjunath BS (2021) Semi supervised segmentation and graph-based tracking of 3d nuclei in time-lapse microscopy. In: 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), pp 385–389. https://doi.org/10.1109/ISBI48211.2021.9433831
Jun B, Ahmadzadegan A, Ardekani A, Solorio L, Vlachos P (2023) Multi-feature-based robust cell tracking. Ann Biomed Eng 51:604–617. https://doi.org/10.1007/s10439-022-03073-1
Türetken E, Wang X, Becker CJ, Haubold C, Fua PV (2017) Network flow integer programming to track elliptical cells in time-lapse sequences. IEEE Trans Med Imaging 36:942–951
Bensch R, Ronneberger O (2015) Cell segmentation and tracking in phase contrast images using graph cut with asymmetric boundary costs. In: 2015 IEEE 12th International Symposium on Biomedical Imaging (ISBI), pp 1220–1223. https://doi.org/10.1109/ISBI.2015.7164093
Dewan MAA, Ahmad MO, Swamy MNS (2011) Tracking biological cells in time-lapse microscopy: An adaptive technique combining motion and topological features. IEEE Trans Biomed Eng 58:1637–1647
Wang Y, Mao H, Yi Z (2019) Stem cell motion-tracking by using deep neural networks with multi-output. Neural Comput Appl 31:3455–3467
Zhou Z, Wang F, Xi W, Chen H, Gao P, He C (2019) Joint multi-frame detection and segmentation for multi-cell tracking. In: Image and Graphics, ICIG 2019, pp 435–446. https://doi.org/10.1007/978-3-030-34110-7_36
Xiao P, Zhong L (2017) Tracking of non-dividing cells by using generalized voronoi diagram. In: 2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp 2684–2687. https://doi.org/10.1109/EMBC.2017.8037410
Vig D, Hamby A, Wolgemuth C (2016) On the quantification of cellular velocity fields. Biophys J 110(7):1469–1475. https://doi.org/10.1016/j.bpj.2016.02.032
Hu T, Huang L, Liu X, Shen H (2019) Real time visual tracking using spatial-aware temporal aggregation network. ArXiv:1908.00692
Reyes-Aldasoro CC, Akerman S, Tozer GM (2008) Measuring the velocity of fluorescently labelled red blood cells with a keyhole tracking algorithm. J Microsc 229:162–173. https://doi.org/10.1111/j.1365-2818.2007.01877.x
Jin L, Zhao F, Lin W, Zhou X, Kuang C, Nedzved A, Ablameyko S, Liu X, Xu Y (2020) Development of fan–shaped tracker for single particle tracking. Microsc Res Tech 83(9):1056–1065. https://doi.org/10.1002/jemt.23496
Yi J, Wu P, Huang Q, Qu H, Hoeppner DJ, Metaxas DN (2019) Online neural cell tracking using blob-seed segmentation and optical flow. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp 1057–1064. https://doi.org/10.1109/CVPRW.2019.00138
Liu K, Lienkamp SS, Shindo A, Wallingford JB, Walz G, Ronneberger O (2014) Optical flow guided cell segmentation and tracking in developing tissue. In: 2014 IEEE 11th International Symposium on Biomedical Imaging (ISBI), pp 298–301. https://doi.org/10.1109/ISBI.2014.6867868
Boukari F, Makrogiannis S (2020) Automated cell tracking using motion prediction-based matching and event handling. IEEE/ACM Trans Comput Biol Bioinform 17(3):959–971. https://doi.org/10.1109/TCBB.2018.2875684
Lee S, Kim H, Higuchi H, Ishikawa M (2021) Visualization method for the cell-level vesicle transport using optical flow and a diverging colormap. Sensors 21(2). https://doi.org/10.3390/s21020522
Sugawara K, Cevrim C, Averof M (2022) Tracking cell lineages in 3d by incremental deep learning. eLife 11:69380
Xie Y, Liu M, Zhou S, Wang Y (2021) A deep local patch matching network for cell tracking in microscopy image sequences without registration. IEEE/ACM Trans Comput Biol Bioinform 19:3202–3212
Löffler K, Scherr T, Mikut R (2021) A graph-based cell tracking algorithm with few manually tunable parameters and automated segmentation error correction. PLoS ONE 16(9):0249257. https://doi.org/10.1371/journal.pone.0249257
Arbelle A, Drayman N, Bray M-A, Alon U, Carpenter AE, Riklin-Raviv T (2015) Analysis of high-throughput microscopy videos: Catching up with cell dynamics. In: MICCAI, vol 9351. https://doi.org/10.1007/978-3-319-24574-4_26
He T, Mao H, Guo J, Yi Z (2017) Cell tracking using deep neural networks with multi-task learning. Image Vis Comput 60:142–153. https://doi.org/10.1016/j.imavis.2016.11.010
Shi J, Lu M (2019) Multiple cell tracking by generalised labelled multi-bernoulli filter. Int J Comput Appl Technol 61(4):273–277. https://doi.org/10.1504/IJCAT.2019.103296
Beard M, Vo B-T, Vo B-N (2020) A solution for large-scale multi-object tracking. IEEE Trans Signal Process 68:2754–2769. https://doi.org/10.1109/TSP.2020.2986136
Nguyen TTD, Shim C, Kim W (2021) Biological cell tracking and lineage inference via random finite sets. In: 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), pp 339–343.https://doi.org/10.1109/ISBI48211.2021.9433957
Belyaev I, Praetorius J-P, Medyukhina A, Figge MT (2021) Enhanced segmentation of label-free cells for automated migration and interaction tracking. Cytometry Part A 99:1218–1229. https://doi.org/10.1002/cyto.a.24466
Ben-Haim, T., Riklin-Raviv, T.: Graph neural network for cell tracking in microscopy videos. In: European Conference on Computer Vision, ECCV 2022, vol. 13681 (2022). https://doi.org/10.1007/978-3-031-19803-8_36
Magnusson KEG, Jalden J, Gilbert PM, Blau HM (2015) Global linking of cell tracks using the viterbi algorithm. IEEE Trans Med Imaging 34(4):911–929. https://doi.org/10.1109/TMI.2014.2370951
Schiegg M, Hanslovsky P, Haubold C, Köthe U, Hufnagel L, Hamprecht FA (2015) Graphical model for joint segmentation and tracking of multiple dividing cells. Bioinformatics 31(6):948–56
Liu M, Liu Y, Qian W, Wang Y (2021) Deepseed local graph matching for densely packed cells tracking. IEEE/ACM Trans Comput Biol Bioinform 18(3):1060–1069. https://doi.org/10.1109/TCBB.2019.2936851
Lu M, Xu B-L, Nener BD, Cong J, Shi J (2022) An accurate cell tracking approach with self-regulated foraging behavior of ant colonies in dynamic microscopy images. Appl Intell 52:1448–1460
Xu B, Lu M, Shi J, Cong J, Nener BD (2021) A joint tracking approach via ant colony evolution for quantitative cell cycle analysis. IEEE J Biomed Health Inf 25(6):2338–2349. https://doi.org/10.1109/JBHI.2020.3032592
Wu D, Xu B, Lu M (2021) A heuristic and reliable track-to-track data association approach for multi-cell track reconstruction. Appl Intell 51:8162–8175
Wang J, Su X, Zhao L, Zhang J (2020) Deep reinforcement learning for data association in cell tracking. Front Bioeng Biotechnol 8:298. https://doi.org/10.3389/fbioe.2020.00298
Chenouard N, Bloch I, Olivo-Marin J-C (2013) Multiple hypothesis tracking for cluttered biological image sequences. IEEE Trans Pattern Anal Mach Intell 35(11):2736–3750. https://doi.org/10.1109/TPAMI.2013.97
Coraluppi SP, Carthel CA (2012) Modified scoring in multiple-hypothesis tracking. J Adv Inf Fusion 7(2):153–164
Coraluppi S, Carthel C, Dickerson SJ, Chiarulli D, Levitan S (2014) Feature-aided multiple-hypothesis tracking and classification of biological cells. In: 17th International Conference on Information Fusion (FUSION), Salamanca, Spain, pp 1–8. https://ieeexplore.ieee.org/document/6916061
Schacherer D, Ritter C, Rohr K (2021) Multiple hypothesis tracking with integrated cell division detection. In: 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), pp 165–168. https://doi.org/10.1109/ISBI48211.2021.9434153
Wen C, Miura T, Voleti V, Yamaguchi K, Tsutsumi M, Yamamoto K, Otomo K, Fujie Y, Teramoto T, Ishihara T, Aoki K, Nemoto T, Hillman EM, Kimura KD (2021) 3deecelltracker, a deep learning-based pipeline for segmenting and tracking cells in 3d time lapse images. eLife 10:59187. https://doi.org/10.7554/eLife.59187
Moen E, Borba E, Miller G, Schwartz M, Bannon D, Koe N, Camplisson I, Kyme D, Pavelchek C, Price T, Kudo T, Pao E, Graf W, Valen DV (2019) Accurate cell tracking and lineage construction in live-cell imaging experiments with deep learning
Zhao M, Liu Q, Jha A, Deng R, Yao T, Mahadevan-Jansen A, Tyska MJ, Millis BA, Huo Y (2021) Voxelembed: 3d instance segmentation and tracking with voxel embedding based deep learning. In: MLMI@MICCAI 2021, vol 12966. https://doi.org/10.1007/978-3-030-87589-3_45
Fujimoto K, Mizugaki T, Rajkumar U, Shigeta H, Seno S, Uchida Y, Ishii M, Bafna V, Matsuda H (2021) A cnn-based cell tracking method for multi-slice intravital imaging data. In: Proceedings of the 12th ACM Conference on Bioinformatics, Computational Biology, and Health Informatics, vol. 35. New York, NY, USA, p 7. https://doi.org/10.1145/3459930.3469559
Dunn KW, Fu C, Ho DJ, Lee S, Han S, Salama P, Delp EJ (2019) Deepsynth: Three-dimensional nuclear segmentation of biological images using neural networks trained with synthetic data. Scientific Reports 9(1):18295. https://doi.org/10.1038/s41598-019-54244-5
Chan S, Huang C, Bai C, Ding W, Chen S (2021) Res2-unext: a novel deep learning framework for few-shot cell image segmentation. Multim Tools Appl 81:13275–13288. https://doi.org/10.1007/s11042-021-10536-5
Newby JM, Schaefer A, Lee P, Forest MG, Lai SK (2018) Convolutional neural networks automate detection for tracking of submicron-scale particles in 2d and 3d. Proc Natl Acad Sci 115(36):9026–9031. https://doi.org/10.1073/pnas.1804420115
Cheng H-J, Hsu C-H, Hung C-L, Lin C-Y (2021) A review for cell and particle tracking on microscopy images using algorithms and deep learning technologies. Biomed J 45:465–471
Lugagne J-B, Lin H, Dunlop MJ (2020) Delta: Automated cell segmentation, tracking, and lineage reconstruction using deep learning. PLoS Comput Biol 16(4). https://doi.org/10.1371/journal.pcbi.1007673
Wu X, Wang W, Yang F, Zhang H, Zuo W (2019) Joint learning of siamese network with top-down modulation and hard example mining for visual tracking. Journal of Electronic Imaging 28:053034. https://doi.org/10.1117/1.JEI.28.5.053034
Li B, Yan J, Wu W, Zhu Z, Hu X (2018) High performance visual tracking with siamese region proposal network. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 8971–8980. https://doi.org/10.1109/CVPR.2018.00935
Kimmel JC, Brack AS, Marshall WF (2021) Deep convolutional and recurrent neural networks for cell motility discrimination and prediction. IEEE/ACM Trans Comput Biol Bioinform 18:562–574
Tasdizen T, Sajjadi M, Javanmardi M, Ramesh N (2018) Improving the robustness of convolutional networks to appearance variability in biomedical images. In: 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), pp 549–553. https://doi.org/10.1109/ISBI.2018.8363636
Moen E, Bannon D, Kudo T, Graf W, Covert MW, Valen DV (2019) Deep learning for cellular image analysis. Nat Methods 16:1233–1246. https://doi.org/10.1038/s41592-019-0403-1
Moghadam MR, Chen YPP (2021) Tracking neutrophil migration in zebrafish model using multi-channel feature learning. IEEE J Biomed Health Inf 25(4):1197–1205. https://doi.org/10.1109/JBHI.2020.3019271
Wu D, Xu B, Lu M, Shi J, Li Z, Guan F, Yang Z (2021) A cell tracking method with deep learning mitosis detection in microscopy images. In: Advances in Swarm Intelligence. ICSI 2021, vol 12690, pp 282–289. https://doi.org/10.1007/978-3-030-78811-7_27
Li R, Gao Q, Rohr K (2021) Multi-object dynamic memory network for cell tracking in time-lapse microscopy images. In: 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), pp 1029–1032. https://doi.org/10.1109/ISBI48211.2021.9433828
Chen Y, Song Y, Zhang C, Zhang F, O’Donnell L, Chrzanowski W, Cai W (2021) Celltrack r-cnn: A novel end-to-end deep neural network for cell segmentation and tracking in microscopy images. In: 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), pp 779–782. https://doi.org/10.1109/ISBI48211.2021.9434057
Marvasti-Zadeh SM, Khaghani J, Cheng L, Ghanei-Yakhdan H, Kasaei S (2021) CHASE: robust visual tracking via cell-level differentiable neural architecture search. In: 32nd British Machine Vision Conference 2021, BMVC 2021, Online, November 22-25, 2021, p 324. https://www.bmvc2021-virtualconference.com/assets/papers/1571.pdf
Xu Y, Šep A, Ban Y, Horaud R, Leal-Taixé L, Alameda-Pineda X (2020) How to train your deep multi-object tracker. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp 6786–6795. https://doi.org/10.1109/CVPR42600.2020.00682
Bise R, Yin Z, Kanade T (2011) Reliable cell tracking by global data association. In: 2011 IEEE International Symposium on Biomedical Imaging: From Nano to Macro, pp 1004–1010. https://doi.org/10.1109/ISBI.2011.5872571
Funding
The research has no funding.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflicts of interest
The authors declare no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Yazdi, R., Khotanlou, H. A survey on automated cell tracking: challenges and solutions. Multimed Tools Appl (2024). https://doi.org/10.1007/s11042-024-18697-9
Received:
Revised:
Accepted:
Published:
DOI: https://doi.org/10.1007/s11042-024-18697-9