An Expert-in-the-loop Paradigm for Learning Medical Image Grouping

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9651)


Image grouping in knowledge-rich domains is challenging, since domain knowledge and expertise are key to transform image pixels into meaningful content. Manually marking and annotating images is not only labor-intensive but also ineffective. Furthermore, most traditional machine learning approaches cannot bridge this gap for the absence of experts’ input. We thus present an interactive machine learning paradigm that allows experts to become an integral part of the learning process. This paradigm is designed for automatically computing and quantifying interpretable grouping of dermatological images. In this way, the computational evolution of an image grouping model, its visualization, and expert interactions form a loop to improve image grouping. In our paradigm, dermatologists encode their domain knowledge about the medical images by grouping a small subset of images via a carefully designed interface. Our learning algorithm automatically incorporates these manually specified connections as constraints for re-organizing the whole image dataset. Performance evaluation shows that this paradigm effectively improves image grouping based on expert knowledge.


Dermatological images Multimodal data Image grouping Visual analytics Interactive machine learning 



This work was partially supported by NIH grant 1R21 LM010039-01A1 and NSF grant IIS-0941452. We would like to thank the participating physicians, the reviewers; and Logical Images, Inc. for images. Any opinions, findings, and conclusions or recommendations expressed in this paper are those of the authors and do not necessarily reflect the official views of the NIH or the NSF.


  1. 1.
    Aronson, A.R.: Effective mapping of biomedical text to the UMLS Metathesaurus: the MetaMap program. In: American Medical Informatics Association Annual Symposium Proceedings, pp. 17–21. AMIA (2001)Google Scholar
  2. 2.
    Belkin, M., Niyogi, P.: Laplacian eigenmaps and spectral techniques for embedding and clustering. NIPS 14, 585–591 (2001)Google Scholar
  3. 3.
    Cai, D., Bao, H., He, X.: Sparse concept coding for visual analysis. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2905–2910. IEEE (2011)Google Scholar
  4. 4.
    Choo, J., Lee, C., Reddy, C.K., Park, H.: UTOPIAN: user-driven topic modeling based on interactive nonnegative matrix factorization. IEEE Trans. Vis. Comput. Graph. 19(12), 1992–2001 (2013)CrossRefGoogle Scholar
  5. 5.
    Choo, J., Lee, H., Kihm, J., Park, H.: iVisClassifier: an interactive visual analytics system for classification based on supervised dimension reduction. In: IEEE Symposium on Visual Analytics Science and Technology (VAST), pp. 27–34. IEEE (2010)Google Scholar
  6. 6.
    Crossno, P.J., Dunlavy, D.M., Shead, T.M.: LSAView: A tool for visual exploration of latent semantic modeling. In: IEEE Symposium on Visual Analytics Science and Technology (VAST), pp. 83–90. IEEE (2009)Google Scholar
  7. 7.
    El-Hamdouchi, A., Willett, P.: Comparison of hierarchic agglomerative clustering methods for document retrieval. Comput. J. 32(3), 220–227 (1989)CrossRefGoogle Scholar
  8. 8.
    Gao, S., Tsang, I.W., Chia, L.-T., Zhao, P.: Local features are not lonely-Laplacian sparse coding for image classification. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3555–3561. IEEE (2010)Google Scholar
  9. 9.
    Guo, X., Li, R., Alm, C., Yu, Q., Pelz, J., Shi, P., Haake, A.: Infusing perceptual expertise, domain knowledge into a human-centered image retrieval system: a prototype application. In: Proceedings of the Symposium on Eye Tracking Research and Applications, pp. 275–278. ACM (2014)Google Scholar
  10. 10.
    Guo, X., Yu, Q., Alm, C.O., Calvelli, C., Pelz, J.B., Shi, P., Haake, A.R.: From spoken narratives to domain knowledge: mining linguistic data for medical image understanding. Artif. Intell. Med. 62(2), 79–90 (2014)CrossRefGoogle Scholar
  11. 11.
    Guo, X., Yu, Q., Li, R., Alm, C.O., Haake, A.R.: Fusing multimodal human expert data to uncover hidden semantics. In: Proceedings of the 7th Workshop on Eye Gaze in Intelligent Human Machine Interaction: Eye-Gaze and Multimodality, pp. 21–26. ACM (2014)Google Scholar
  12. 12.
    Holzinger, A.: Human-Computer Interaction and Knowledge Discovery (HCI-KDD): what is the benefit of bringing those two fields to work together? In: Cuzzocrea, A., Kittl, C., Simos, D.E., Weippl, E., Xu, L. (eds.) CD-ARES 2013. LNCS, vol. 8127, pp. 319–328. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  13. 13.
    Hu, Y., Boyd-Graber, J., Satinoff, B., Smith, A.: Interactive topic modeling. Mach. Learn. 95(3), 423–469 (2014)MathSciNetCrossRefGoogle Scholar
  14. 14.
    Lee, H., Battle, A., Raina, R., Ng, A.Y.: Efficient sparse coding algorithms. In: Advances in Neural Information Processing Systems, pp. 801–808 (2006)Google Scholar
  15. 15.
    Lee, H., Kihm, J., Choo, J., Stasko, J., Park, H.: iVisClustering: an interactive visual document clustering via topic modeling. In: Computer Graphics Forum, vol. 31, pp. 1155–1164. Wiley Online Library (2012)Google Scholar
  16. 16.
    Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60(2), 91–110 (2004)CrossRefGoogle Scholar
  17. 17.
    McCoy, W., Alm, C.O., Calvelli, C., Li, R., Pelz, J.B., Shi, P., Haake, A.: Annotation schemes to encode domain knowledge in medical narratives. In: Ide, N., Xia, F. (eds.) Proceedings of the 6th Linguistic Annotation Workshop, pp. 95–103. Association for Computational Linguistics, Stroudsburg (2012)Google Scholar
  18. 18.
    Vaidyanathan, P., Pelz, J., Li, R., Mulpuru, S., Wang, D., Shi, P., Calvelli, C., Haake, A.: Using human experts’ gaze data to evaluate image processing algorithms. In: Hemami, S., Pappas, T.N. (eds.) \(10^{th}\) IVMSP Workshop: Perception and Visual Signal Analysis, pp. 129–134. IEEE (2011)Google Scholar
  19. 19.
    Van der Maaten, L., Hinton, G.: Visualizing data using t-SNE. J. Mach. Learn. Res. 9(2579–2605), 85 (2008)zbMATHGoogle Scholar
  20. 20.
    Zheng, M., Bu, J., Chen, C., Wang, C., Zhang, L., Qiu, G., Cai, D.: Graph regularized sparse coding for image representation. IEEE Trans. Image Process. 20(5), 1327–1336 (2011)MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  1. 1.B. Thomas Golisano College of Computing and Information SciencesRochesterUSA
  2. 2.Rochester Institute of TechnologyRochesterUSA

Personalised recommendations