Advertisement

Unsupervised Domain Adaptation for Semantic Segmentation of NIR Images Through Generative Latent Search

Conference paper
  • 808 Downloads
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12351)

Abstract

Segmentation of the pixels corresponding to human skin is an essential first step in multiple applications ranging from surveillance to heart-rate estimation from remote-photoplethysmography. However, the existing literature considers the problem only in the visible-range of the EM-spectrum which limits their utility in low or no light settings where the criticality of the application is higher. To alleviate this problem, we consider the problem of skin segmentation from the Near-infrared images. However, Deep learning based state-of-the-art segmentation techniques demands large amounts of labelled data that is unavailable for the current problem. Therefore we cast the skin segmentation problem as that of target-independent Unsupervised Domain Adaptation (UDA) where we use the data from the Red-channel of the visible-range to develop skin segmentation algorithm on NIR images. We propose a method for target-independent segmentation where the ‘nearest-clone’ of a target image in the source domain is searched and used as a proxy in the segmentation network trained only on the source domain. We prove the existence of ‘nearest-clone’ and propose a method to find it through an optimization algorithm over the latent space of a Deep generative model based on variational inference. We demonstrate the efficacy of the proposed method for NIR skin segmentation over the state-of-the-art UDA segmentation methods on the two newly created skin segmentation datasets in NIR domain despite not having access to the target NIR data. Additionally, we report state-of-the-art results for adaption from Synthia to Cityscapes which is a popular setting in Unsupervised Domain Adaptation for semantic segmentation. The code and datasets are available at https://github.com/ambekarsameer96/GLSS.

Keywords

Unsupervised domain adaptation Semantic segmentation Near IR dataset VAE 

Supplementary material

504443_1_En_25_MOESM1_ESM.pdf (1.5 mb)
Supplementary material 1 (pdf 1500 KB)

References

  1. 1.
    CyCADA: cycle consistent adversarial domain adaptation. In: International Conference on Machine Learning (ICML) (2018)Google Scholar
  2. 2.
    Al-Mohair, H.K., Saleh, J., Saundi, S.: Impact of color space on human skin color detection using an intelligent system. In: 1st WSEAS International Conference on Image Processing and Pattern Recognition (IPPR 2013), vol. 2 (2013)Google Scholar
  3. 3.
    Brancati, N., De Pietro, G., Frucci, M., Gallo, L.: Human skin detection through correlation rules between the YCb and YCr subspaces based on dynamic color clustering. Comput. Vis. Image Underst. 155, 33–42 (2017)CrossRefGoogle Scholar
  4. 4.
    Chang, W.L., Wang, H.P., Peng, W.H., Chiu, W.C.: All about structure: adapting structural information across domains for boosting semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1900–1909 (2019)Google Scholar
  5. 5.
    Chaurasia, A., Culurciello, E.: LinkNet: exploiting encoder representations for efficient semantic segmentation. In: 2017 IEEE Visual Communications and Image Processing (VCIP), pp. 1–4. IEEE (2017)Google Scholar
  6. 6.
    Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F., Adam, H.: Encoder-decoder with atrous separable convolution for semantic image segmentation. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11211, pp. 833–851. Springer, Cham (2018).  https://doi.org/10.1007/978-3-030-01234-2_49CrossRefGoogle Scholar
  7. 7.
    Chen, W., Wang, K., Jiang, H., Li, M.: Skin color modeling for face detection and segmentation: a review and a new approach. Multimedia Tools Appl. 75(2), 839–862 (2016).  https://doi.org/10.1007/s11042-014-2328-0CrossRefGoogle Scholar
  8. 8.
    Chen, W.C., Wang, M.S.: Region-based and content adaptive skin detection in color images. Int. J. Pattern Recogn. Artif. Intell. 21(05), 831–853 (2007)CrossRefGoogle Scholar
  9. 9.
    Chollet, F.: Xception: deep learning with depthwise separable convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1251–1258 (2017)Google Scholar
  10. 10.
    Cordts, M., et al.: The cityscapes dataset for semantic urban scene understanding. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3213–3223 (2016)Google Scholar
  11. 11.
    Cover, T., Hart, P.: Nearest neighbor pattern classification. IEEE Trans. Inf. Theory 13(1), 21–27 (1967)zbMATHCrossRefGoogle Scholar
  12. 12.
    Dourado, A., Guth, F., de Campos, T.E., Weigang, L.: Domain adaptation for holistic skin detection. arXiv preprint arXiv:1903.06969 (2019)
  13. 13.
    Erdem, C., Ulukaya, S., Karaali, A., Erdem, A.T.: Combining Haar feature and skin color based classifiers for face detection. In: 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1497–1500. IEEE (2011)Google Scholar
  14. 14.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)Google Scholar
  15. 15.
    He, Y., et al.: Semi-supervised skin detection by network with mutual guidance. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2111–2120 (2019)Google Scholar
  16. 16.
    Hoffman, J., Wang, D., Yu, F., Darrell, T.: FCNs in the wild: pixel-level adversarial and constraint-based adaptation. arXiv preprint arXiv:1612.02649 (2016)
  17. 17.
    Hore, A., Ziou, D.: Image quality metrics: PSNR vs. SSIM. In: 2010 20th International Conference on Pattern Recognition, pp. 2366–2369. IEEE (2010)Google Scholar
  18. 18.
    Hsu, R.L., Abdel-Mottaleb, M., Jain, A.K.: Face detection in color images. IEEE Trans. Pattern Anal. Mach. Intell. 24(5), 696–706 (2002)CrossRefGoogle Scholar
  19. 19.
    Huynh-Thu, Q., Meguro, M., Kaneko, M.: Skin-color-based image segmentation and its application in face detection. In: MVA, pp. 48–51 (2002)Google Scholar
  20. 20.
    Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 694–711. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46475-6_43CrossRefGoogle Scholar
  21. 21.
    Jones, M.J., Rehg, J.M.: Statistical color models with application to skin detection. Int. J. Comput. Vis. 46(1), 81–96 (2002).  https://doi.org/10.1023/A:1013200319198zbMATHCrossRefGoogle Scholar
  22. 22.
    Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  23. 23.
    Kingma, D.P., Welling, M.: Auto-encoding variational Bayes. arXiv preprint arXiv:1312.6114 (2013)
  24. 24.
    Kong, S.G., Heo, J., Abidi, B.R., Paik, J., Abidi, M.A.: Recent advances in visual and infrared face recognition-a review. Comput. Vis. Image Underst. 97(1), 103–135 (2005)CrossRefGoogle Scholar
  25. 25.
    Kovac, J., Peer, P., Solina, F.: Human skin color clustering for face detection, vol. 2. IEEE (2003)Google Scholar
  26. 26.
    Li, Y., Yuan, L., Vasconcelos, N.: Bidirectional learning for domain adaptation of semantic segmentation. arXiv preprint arXiv:1904.10620 (2019)
  27. 27.
    Lin, T.Y., Dollár, P., Girshick, R., He, K., Hariharan, B., Belongie, S.: Feature pyramid networks for object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2117–2125 (2017)Google Scholar
  28. 28.
    Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal loss for dense object detection. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2980–2988 (2017)Google Scholar
  29. 29.
    Liu, Q., Peng, G.: A robust skin color based face detection algorithm. In: 2010 2nd International Asia Conference on Informatics in Control, Automation and Robotics (CAR 2010), vol. 2, pp. 525–528. IEEE (2010)Google Scholar
  30. 30.
    Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3431–3440 (2015)Google Scholar
  31. 31.
    Luo, Y., Zheng, L., Guan, T., Yu, J., Yang, Y.: Taking a closer look at domain shift: category-level adversaries for semantics consistent domain adaptation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2507–2516 (2019)Google Scholar
  32. 32.
    Mahmoodi, M.R.: High performance novel skin segmentation algorithm for images with complex background. arXiv preprint arXiv:1701.05588 (2017)
  33. 33.
    Mahmoodi, M.R., Sayedi, S.M.: A comprehensive survey on human skin detection. Int. J. Image Graph. Sig. Process. 8(5), 1–35 (2016)CrossRefGoogle Scholar
  34. 34.
    Moallem, P., Mousavi, B.S., Monadjemi, S.A.: A novel fuzzy rule base system for pose independent faces detection. Appl. Soft Comput. 11(2), 1801–1810 (2011)CrossRefGoogle Scholar
  35. 35.
    Pan, Z., Healey, G., Prasad, M., Tromberg, B.: Face recognition in hyperspectral images. IEEE Trans. Pattern Anal. Mach. Intell. 25(12), 1552–1560 (2003)CrossRefGoogle Scholar
  36. 36.
    Pandey, P., Prathosh, A.P., Kyatham, V., Mishra, D., Dastidar, T.R.: Target-independent domain adaptation for WBC classification using generative latent search. arXiv preprint arXiv:2005.05432 (2020)
  37. 37.
    Pandey, P., Prathosh, A., Kohli, M., Pritchard, J.: Guided weak supervision for action recognition with scarce data to assess skills of children with autism. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 463–470 (2020)Google Scholar
  38. 38.
    Prathosh, A., Praveena, P., Mestha, L.K., Bharadwaj, S.: Estimation of respiratory pattern from video using selective ensemble aggregation. IEEE Trans. Sig. Process. 65(11), 2902–2916 (2017)MathSciNetzbMATHCrossRefGoogle Scholar
  39. 39.
    Qiang-rong, J., Hua-lan, L.: Robust human face detection in complicated color images. In: 2010 2nd IEEE International Conference on Information Management and Engineering, pp. 218–221. IEEE (2010)Google Scholar
  40. 40.
    Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015).  https://doi.org/10.1007/978-3-319-24574-4_28CrossRefGoogle Scholar
  41. 41.
    Ros, G., Sellart, L., Materzynska, J., Vazquez, D., Lopez, A.M.: The synthia dataset: a large collection of synthetic images for semantic segmentation of urban scenes. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3234–3243 (2016)Google Scholar
  42. 42.
    Seow, M.J., Valaparla, D., Asari, V.K.: Neural network based skin color model for face detection. In: 2003 Proceedings of the 32nd Applied Imagery Pattern Recognition Workshop, pp. 141–145. IEEE (2003)Google Scholar
  43. 43.
    Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2818–2826 (2016)Google Scholar
  44. 44.
    Tan, M., Le, Q.V.: EfficientNet: rethinking model scaling for convolutional neural networks. arXiv preprint arXiv:1905.11946 (2019)
  45. 45.
    Taqa, A.Y., Jalab, H.A.: Increasing the reliability of skin detectors. Sci. Res. Essays 5(17), 2480–2490 (2010)Google Scholar
  46. 46.
    Tsai, Y.H., Hung, W.C., Schulter, S., Sohn, K., Yang, M.H., Chandraker, M.: Learning to adapt structured output space for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7472–7481 (2018)Google Scholar
  47. 47.
    Vu, T.H., Jain, H., Bucher, M., Cord, M., Pérez, P.: ADVENT: adversarial entropy minimization for domain adaptation in semantic segmentation. In: CVPR (2019)Google Scholar
  48. 48.
    Vu, T.H., Jain, H., Bucher, M., Cord, M., Pérez, P.: DADA: depth-aware domain adaptation in semantic segmentation. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 7364–7373 (2019)Google Scholar
  49. 49.
    Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)CrossRefGoogle Scholar
  50. 50.
    Wu, Q., Cai, R., Fan, L., Ruan, C., Leng, G.: Skin detection using color processing mechanism inspired by the visual system (2012)Google Scholar
  51. 51.
    Zaidan, A., Ahmad, N.N., Karim, H.A., Larbani, M., Zaidan, B., Sali, A.: On the multi-agent learning neural and Bayesian methods in skin detector and pornography classifier: an automated anti-pornography system. Neurocomputing 131, 397–418 (2014)CrossRefGoogle Scholar
  52. 52.
    Zhao, H., Shi, J., Qi, X., Wang, X., Jia, J.: Pyramid scene parsing network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2881–2890 (2017)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Indian Institute of Technology DelhiNew DelhiIndia

Personalised recommendations