Human Re-Identification with a Robot Thermal Camera Using Entropy-Based Sampling

Abstract

Human re-identification is an important feature of domestic service robots, in particular for elderly monitoring and assistance, because it allows them to perform personalized tasks and human-robot interactions. However vision-based re- identification systems are subject to limitations due to human pose and poor lighting conditions. This paper presents a new re-identification method for service robots using thermal images. In robotic applications, as the number and size of thermal datasets is limited, it is hard to use approaches that require huge amount of training samples. We propose a re-identification system that can work using only a small amount of data. During training, we perform entropy-based sampling to obtain a thermal dictionary for each person. Then, a symbolic representation is produced by converting each video into sequences of dictionary elements. Finally, we train a classifier using this symbolic representation and geometric distribution within the new representation domain. The experiments are performed on a new thermal dataset for human re-identification, which includes various situations of human motion, poses and occlusion, and which is made publicly available for research purposes. The proposed approach has been tested on this dataset and its improvements over standard approaches have been demonstrated.

References

  1. 1.

    Barbosa, I.B., Cristani, M., Del Bue, A., Bazzani, L., Murino, V.: Re-Identification with Rgb-D Sensors. In: European Conference on Computer Vision, pp. 433–442. Springer (2012)

  2. 2.

    Bedagkar-Gala, A., Shah, S.K.: A survey of approaches and trends in person re-identification. Image Vis. Comput. 32(4), 270–286 (2014). https://doi.org/10.1016/j.imavis.2014.02.001

    Article  Google Scholar 

  3. 3.

    Bellotto, N., Hu, H.: A bank of unscented kalman filters for multimodal human perception with mobile service robots. Int. J. Soc. Robot. 2(2), 121–136 (2010). https://doi.org/10.1007/s12369-010-0047-x

    Article  Google Scholar 

  4. 4.

    Chang, C.C., Lin, C.J.: LIBSVM: A library for support vector machines. ACM Trans. Intell. Syst. Technol. 2, 27:1–27:27 (2011). Software available at http://www.csie.ntu.edu.tw/cjlin/libsvm

    Article  Google Scholar 

  5. 5.

    Chen, D., Yuan, Z., Hua, G., Zheng, N., Wang, J.: Similarity Learning on an Explicit Polynomial Kernel Feature Map for Person Re-Identification. In: IEEE CVPR, pp. 1565–1573. https://doi.org/10.1109/CVPR.2015.7298764 (2015)

  6. 6.

    Cho, Y.J., Yoon, K.J.: Improving Person Re-Identification via Pose-Aware Multi-Shot Matching. In: 2016 IEEE CVPR, pp. 1354–1362. https://doi.org/10.1109/CVPR.2016.151 (2016)

  7. 7.

    Choi, J., Hu, S., Young, S.S., Davis, L.S.: Thermal to Visible Face Recognition. In: SPIE DSS-DS107: Biometric Technology for Human Identification IX (2012)

  8. 8.

    Coşar, S., Coppola, C., Bellotto, N.: Volume-based human re-identification with RGB-d cameras. In: Proceedings of the 12th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP) - Volume 4: VISAPP, pp. 389–397 (2017)

  9. 9.

    Cortes, C., Vapnik, V.: Support-vector networks. Mach. Learn. 20(3), 273–297 (1995). https://doi.org/10.1023/A:1022627411411

    MATH  Google Scholar 

  10. 10.

    Farenzena, M., Bazzani, L., Perina, A., Murino, V., Cristani, M.: Person Re-Identification by Symmetry-Driven Accumulation of Local Features. In: IEEE CVPR, pp. 2360–2367. https://doi.org/10.1109/CVPR.2010.5539926 (2010)

  11. 11.

    Ghiass, R.S., Arandjelović, O., Bendada, H., Maldague, X.: Infrared Face Recognition: a Literature Review. In: The 2013 International Joint Conference on Neural Networks (IJCNN), pp. 1–10. https://doi.org/10.1109/IJCNN.2013.6707096 (2013)

  12. 12.

    Hermosilla, G., del Solar, J.R., Verschae, R., Correa, M.: A comparative study of thermal face recognition methods in unconstrained environments. Pattern Recogn. 45(7), 2445–2459 (2012). https://doi.org/10.1016/j.patcog.2012.01.001

    Article  Google Scholar 

  13. 13.

    Joachims, T. Nédellec, C., Rouveirol, C. (eds.): Text Categorization with Support Vector Machines: Learning with Many Relevant Features. Springer, Berlin (1998)

  14. 14.

    Koide, K., Miura, J.: Identification of a specific person using color, height, and gait features for a person following robot. Robot. Auton. Syst. 84, 76–87 (2016). https://doi.org/10.1016/j.robot.2016.07.004

    Article  Google Scholar 

  15. 15.

    Kullback, S., Leibler, R.A.: On information and sufficiency. Ann. Math. Statist. 22(1), 79–86 (1951). https://doi.org/10.1214/aoms/1177729694

    MathSciNet  Article  Google Scholar 

  16. 16.

    Kviatkovsky, I., Adam, A., Rivlin, E.: Color invariants for person reidentification. IEEE Trans. Pattern Anal. Mach. Intell. 35(7), 1622–1634 (2013). https://doi.org/10.1109/TPAMI.2012.246

    Article  Google Scholar 

  17. 17.

    Li, W., Zhao, R., Xiao, T., Wang, X.: Deepreid: Deep Filter Pairing Neural Network for Person Re-Identification. In: IEEE CVPR (2014)

  18. 18.

    Mensink, T., Verbeek, J., Perronnin, F., Csurka, G.: Distance-based image classification: Generalizing to new classes at near-zero cost. IEEE Trans. Pattern Anal. Mach. Intell. 35(11), 2624–2637 (2013). https://doi.org/10.1109/TPAMI.2013.83

    Article  Google Scholar 

  19. 19.

    Møgelmose, A., Bahnsen, C., Moeslund, T.B., Clapės, A., Escalera, S.: Tri-Modal Person Re-Identification with Rgb, Depth and Thermal Features. In: IEEE CVPR Workshops 2013, Portland, pp. 301–307 (2013)

  20. 20.

    Munaro, M., Basso, A., Fossati, A., Gool, L.V.: Menegatti, E.: 3D Reconstruction of Freely Moving Persons for Re-Identification with a Depth Sensor. In: 2014 IEEE ICRA, pp. 4512–4519. https://doi.org/10.1109/ICRA.2014.6907518 (2014)

  21. 21.

    Munaro, M., Fossati, A., Basso, A, Menegatti, E., Van Gool, L.: One-shot person re-identification with a consumer depth camera. In: Gong, S., Cristani, M., Yan, S., Loy, C.C. (eds.) Person Re-Identification, pp 161–181. Springer, London (2014). https://doi.org/10.1007/978-1-4471-6296-4_8

  22. 22.

    Nguyen, D.T., Hong, H.G., Kim, K.W., Park, K.R.: Person recognition system based on a combination of body images from visible light and thermal cameras. Sensors 17, 605 (2017)

    Article  Google Scholar 

  23. 23.

    Olesen, B.W.: Thermal comfort. Techn. Rev. 2, 3–37 (1982)

    Google Scholar 

  24. 24.

    Paisitkriangkrai, S., Shen, C., van den Hengel, A.: Learning to Rank in Person Re-Identification with Metric Ensembles. In: IEEE CVPR (2015)

  25. 25.

    Pala, F., Satta, R., Fumera, G., Roli, F.: Multimodal person reidentification using rgb-d cameras. IEEE Trans. Circ. Syst. Video Technol. 26(4), 788–799 (2016)

    Article  Google Scholar 

  26. 26.

    Vezzani, R., Baltieri, D., Cucchiara, R.: People reidentification in surveillance and forensics: A survey. ACM Comput. Surv. 46(2), 29:1–29:37 (2013)

    Article  Google Scholar 

  27. 27.

    W, B.: The temperature of the skin surface. J. Am. Med. Assoc. 106(14), 1158–1162 (1936). https://doi.org/10.1001/jama.1936.02770140020007x

    Article  Google Scholar 

  28. 28.

    Wang, X., Doretto, G., Sebastian, T., Rittscher, J., Tu, P.: Shape and Appearance Context Modeling. In: IN: PROC. ICCV (2007)

  29. 29.

    Weinrich, C., Volkhardt, M., Gross, H.M.: Appearance-Based 3D Upper-Body Pose Estimation and Person Re-Identification on Mobile Robots. In: 2013 IEEE International Conference on Systems, Man, and Cybernetics, pp. 4384–4390. https://doi.org/10.1109/SMC.2013.748 (2013)

  30. 30.

    Wengefeld, T., Eisenbach, M., Trinh, T.Q., Gross, H.M.: May i be your personal coach? bringing together person tracking and visual re-identification on a mobile robot. ISR 2016 (2016)

  31. 31.

    Wu, Z., Peng, M., Chen, T.: Thermal Face Recognition Using Convolutional Neural Network. In: 2016 International Conference on Optoelectronics and Image Processing (ICOIP), pp. 6–9. https://doi.org/10.1109/OPTIP.2016.7528489 (2016)

  32. 32.

    Yan, Z., Duckett, T., Bellotto, N.: Online learning for human classification in 3d lidar-based tracking. In: Proceedings of the 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (2017)

  33. 33.

    Zhao, Y., Zhao, X., Xiang, Z., Liu, Y.: Online learning of dynamic multi-view gallery for person re-identification. Multimed. Tools Appl. 76(1), 217–241 (2017). https://doi.org/10.1007/s11042-015-3015-5

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by the EU H2020 project “ENRICHME” (grant agreement nr. 643691).

Author information

Affiliations

Authors

Corresponding author

Correspondence to Serhan Coşar.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Coşar, S., Bellotto, N. Human Re-Identification with a Robot Thermal Camera Using Entropy-Based Sampling. J Intell Robot Syst 98, 85–102 (2020). https://doi.org/10.1007/s10846-019-01026-w

Download citation

Keywords

  • Service robots
  • Re-identification
  • Elderly care
  • Thermal camera
  • Occlusion
  • Body motion