Feature Learning Based on Visual Similarity Triplets in Medical Image Analysis: A Case Study of Emphysema in Chest CT Scans

  • Silas Nyboe ØrtingEmail author
  • Jens Petersen
  • Veronika Cheplygina
  • Laura H. Thomsen
  • Mathilde M. W. Wille
  • Marleen de Bruijne
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11043)


Supervised feature learning using convolutional neural networks (CNNs) can provide concise and disease relevant representations of medical images. However, training CNNs requires annotated image data. Annotating medical images can be a time-consuming task and even expert annotations are subject to substantial inter- and intra-rater variability. Assessing visual similarity of images instead of indicating specific pathologies or estimating disease severity could allow non-experts to participate, help uncover new patterns, and possibly reduce rater variability. We consider the task of assessing emphysema extent in chest CT scans. We derive visual similarity triplets from visually assessed emphysema extent and learn a low dimensional embedding using CNNs. We evaluate the networks on 973 images, and show that the CNNs can learn disease relevant feature representations from derived similarity triplets. To our knowledge this is the first medical image application where similarity triplets has been used to learn a feature representation that can be used for embedding unseen test images.


Feature learning Similarity triplets Emphysema assessment 


  1. 1.
    Chollet, F., et al.: Keras (2015).
  2. 2.
    COPDGene CT Workshop Group, Graham Barr, R., et al.: A combined pulmonary-radiology workshop for visual evaluation of COPD: study design, chest CT findings and concordance with quantitative evaluation. COPD J. Chronic Obstr. Pulm. Dis. 9(2), 151–159 (2012)Google Scholar
  3. 3.
    Goffin, R.D., Olson, J.M.: Is it all relative? Comparative judgments and the possible improvement of self-ratings and ratings of others. Perspect. Psychol. Sci. 6(1), 48–60 (2011)CrossRefGoogle Scholar
  4. 4.
    Global Strategy for the Diagnosis, Management and Prevention of COPD, Global Initiative for Chronic Obstructive Lung Disease (GOLD) (2017)Google Scholar
  5. 5.
    Jones, I., Wheadon, C.: Peer assessment using comparative and absolute judgement. Stud. Educ. Eval. 47, 93–101 (2015)CrossRefGoogle Scholar
  6. 6.
    Ørting, S.N., Cheplygina, V., Petersen, J., Thomsen, L.H., Wille, M.M.W., de Bruijne, M.: Crowdsourced emphysema assessment. In: Cardoso, M.J., et al. (eds.) LABELS/CVII/STENT -2017. LNCS, vol. 10552, pp. 126–135. Springer, Cham (2017). Scholar
  7. 7.
    Pedersen, J.H., et al.: The Danish randomized lung cancer CT screening trial-overall design and results of the prevalence round. J. Thorac. Oncol. 4(5), 608–614 (2009)CrossRefGoogle Scholar
  8. 8.
    Schroff, F., Kalenichenko, D., Philbin, J.: Facenet: a unified embedding for face recognition and clustering. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 815–823 (2015)Google Scholar
  9. 9.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition (2014). arXiv preprint: arXiv:1409.1556
  10. 10.
    Veit, A., Belongie, S., Karaletsos, T.: Conditional similarity networks. In: Computer Vision and Pattern Recognition (CVPR 2017) (2017)Google Scholar
  11. 11.
    Wagner, S.H., Goffin, R.D.: Differences in accuracy of absolute and comparative performance appraisal methods. Organ. Behav. Hum. Decis. Process. 70(2), 95–103 (1997)CrossRefGoogle Scholar
  12. 12.
    Wah, C., Van Horn, G., Branson, S., Maji, S., Perona, P., Belongie, S.: Similarity comparisons for interactive fine-grained categorization. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 859–866 (2014)Google Scholar
  13. 13.
    Wang, J., et al.: Learning fine-grained image similarity with deep ranking. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1386–1393 (2014)Google Scholar
  14. 14.
    Wilber, M.J., Kwak, I.S., Belongie, S.J.: Cost-effective hits for relative similarity comparisons. In: Second AAAI Conference on Human Computation and Crowdsourcing (2014)Google Scholar
  15. 15.
    Wille, M.M., Thomsen, L.H., Dirksen, A., Petersen, J., Pedersen, J.H., Shaker, S.B.: Emphysema progression is visually detectable in low-dose CT in continuous but not in former smokers. Eur. Radiol. 24(11), 2692–2699 (2014)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Silas Nyboe Ørting
    • 1
    Email author
  • Jens Petersen
    • 1
  • Veronika Cheplygina
    • 2
  • Laura H. Thomsen
    • 3
  • Mathilde M. W. Wille
    • 4
  • Marleen de Bruijne
    • 1
    • 5
  1. 1.Department of Computer ScienceUniversity of CopenhagenCopenhagenDenmark
  2. 2.Medical Image Analysis (IMAG/e), Department of Biomedical EngineeringEindhoven University of TechnologyEindhovenThe Netherlands
  3. 3.Department of Internal MedicineHvidovre HospitalCopenhagenDenmark
  4. 4.Department of Diagnostic ImagingBispebjerg HospitalCopenhagenDenmark
  5. 5.Biomedical Imaging Group Rotterdam, Departments of Radiology and Medical InformaticsErasmus MC - University Medical Center RotterdamRotterdamThe Netherlands

Personalised recommendations