Advertisement

Online learning for 3D LiDAR-based human detection: experimental analysis of point cloud clustering and classification methods

  • Zhi YanEmail author
  • Tom Duckett
  • Nicola Bellotto
Article
  • 53 Downloads

Abstract

This paper presents a system for online learning of human classifiers by mobile service robots using 3D LiDAR sensors, and its experimental evaluation in a large indoor public space. The learning framework requires a minimal set of labelled samples (e.g. one or several samples) to initialise a classifier. The classifier is then retrained iteratively during operation of the robot. New training samples are generated automatically using multi-target tracking and a pair of “experts” to estimate false negatives and false positives. Both classification and tracking utilise an efficient real-time clustering algorithm for segmentation of 3D point cloud data. We also introduce a new feature to improve human classification in sparse, long-range point clouds. We provide an extensive evaluation of our the framework using a 3D LiDAR dataset of people moving in a large indoor public space, which is made available to the research community. The experiments demonstrate the influence of the system components and improved classification of humans compared to the state-of-the-art.

Keywords

Online learning Human detection Point cloud segmentation 3D LiDAR-based tracking Dataset 

Mathematics Subject Classification

68T40 93C85 

Notes

References

  1. Arras, K. O., Mozos, O. M., & Burgard, W. (2007). Using boosted features for the detection of people in 2d range data. In Proceedings of the 2007 IEEE international conference on robotics and automation (ICRA) (pp. 3402–3407).Google Scholar
  2. Bellotto, N., & Hu, H. (2009). Multisensor-based human detection and tracking for mobile service robots. IEEE Transactions on Systems, Man, and Cybernetics Part B, 39(1), 167–181.CrossRefGoogle Scholar
  3. Bellotto, N., & Hu, H. (2010). Computationally efficient solutions for tracking people with a mobile robot: An experimental evaluation of bayesian filters. Autonomous Robots, 28, 425–438.CrossRefGoogle Scholar
  4. Beyer, L., Hermans, A., Linder, T., Arras, K. O., & Leibe, B. (2018). Deep person detection in two-dimensional range data. IEEE Robotics and Automation Letters, 3(3), 2726–2733.CrossRefGoogle Scholar
  5. Bianco, S., Ciocca, G., Napoletano, P., & Schettini, R. (2015). An interactive tool for manual, semi-automatic and automatic video annotation. Computer Vision and Image Understanding, 131, 88–99.CrossRefGoogle Scholar
  6. Bogoslavskyi, I., & Stachniss, C. (2016) Fast range image-based segmentation of sparse 3d laser scans for online operation. In Proceedings of the 2016 IEEE/RSJ international conference on intelligent robots and systems (IROS) (pp. 163–169).Google Scholar
  7. Chang, C. C., & Lin, C. J. (2011). LIBSVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2, 1–27.CrossRefGoogle Scholar
  8. Cortes, C., & Vapnik, V. (1995). Support-vector networks. Machine Learning, 20(3), 273–297.zbMATHGoogle Scholar
  9. Dequaire, J., Ondruska, P., Rao, D., Wang, D., & Posner, I. (2017). Deep tracking in the wild: End-to-end tracking using recurrent neural networks. International Journal of Robotics Research (IJRR), 37, 1–21.Google Scholar
  10. Deuge, M.D., Quadros, A., Hung, C., & Douillard, B. (2013). Unsupervised feature learning for classification of outdoor 3d scans. In Proceedings of the Australasian conference on robotics and automation (ACRA).Google Scholar
  11. Dewan, A., Caselitz, T., Tipaldi, G.D., & Burgard, W. (2016). Motion-based detection and tracking in 3D LiDAR scans. In Proceedings of the 2016 IEEE international conference on robotics and automation (ICRA) (pp. 4508–4513).Google Scholar
  12. Dondrup, C., Bellotto, N., Jovan, F., & Hanheide, M. (2015). Real-time multisensor people tracking for human-robot spatial interaction. In Proceedings of the 2015 IEEE international conference on robotics and automation (ICRA), workshop on machine learning for social robotics.Google Scholar
  13. Douillard, B., Underwood, J. P., Kuntz, N., Vlaskine, V., Quadros, A. J., Morton, P., & Frenkel, A. (2011). On the segmentation of 3d LIDAR point clouds. In Proceedings of the 2011 IEEE international conference on robotics and automation (ICRA) (pp. 2798–2805).Google Scholar
  14. Everingham, M., Gool, L. J. V., Williams, C. K. I., Winn, J. M., & Zisserman, A. (2010). The pascal visual object classes (VOC) challenge. International Journal of Computer Vision, 88, 303–338.CrossRefGoogle Scholar
  15. Geiger, A., Lenz, P., & Urtasun, R. (2012). Are we ready for autonomous driving? the KITTI vision benchmark suite. In Proceedings of the 2012 IEEE conference on computer vision and pattern recognition (CVPR) (pp. 3354–3361).Google Scholar
  16. González, A., Villalonga, G., Xu, J., Vázquez, D., Amores, J., & López, A.M. (2015). Multiview random forest of local experts combining RGB and LIDAR data for pedestrian detection. In Proceedings of the 2015 IEEE intelligent vehicles symposium (IV) (pp. 356–361).Google Scholar
  17. Häselich, M., Jöbgen, B., Wojke, N., Hedrich, J., & Paulus, D. (2014). Confidence-based pedestrian tracking in unstructured environments using 3d laser distance measurements. In Proceedings of the 2014 IEEE/RSJ international conference on intelligent robots and systems (IROS) (pp. 4118–4123).Google Scholar
  18. Jafari, O. H., Mitzel, D., & Leibe, B. (2014). Real-time RGB-D based people detection and tracking for mobile robots and head-worn cameras. In Proceedings of the 2014 IEEE international conference on robotics and automation (ICRA) (pp. 5636–5643).Google Scholar
  19. Julier, S. J., & Uhlmann, J. K. (2004). Unscented filtering and nonlinear estimation. Proceedings of the IEEE, 92(3), 401–422.CrossRefGoogle Scholar
  20. Kalal, Z., Mikolajczyk, K., & Matas, J. (2012). Tracking-learning-detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 34, 1409–1422.CrossRefGoogle Scholar
  21. Kavasidis, I., Palazzo, S., Salvo, R. D., Giordano, D., & Spampinato, C. (2012). A semi-automatic tool for detection and tracking ground truth generation in videos. In Proceedings of the 1st international workshop on visual interfaces for ground truth collection in computer vision applications.Google Scholar
  22. Keerthi, S. S., & Lin, C. J. (2003). Asymptotic behaviors of support vector machines with gaussian kernel. Neural Computation, 15(7), 1667–1689.CrossRefzbMATHGoogle Scholar
  23. Kidono, K., Miyasaka, T., Watanabe, A., Naito, T., & Miura, J. (2011). Pedestrian recognition using high-definition LIDAR. In Proceedings of the 2011 IEEE intelligent vehicles symposium (IV) (pp. 405–410).Google Scholar
  24. Krajnik, T., Fentanes, J. P., Santos, J. M., & Duckett, T. (2017). Fremen: Frequency map enhancement for long-term mobile robot autonomy in changing environments. IEEE Transactions on Robotics, 33, 964–977.CrossRefGoogle Scholar
  25. Leigh, A., Pineau, J., Olmedo, N. A., & Zhang, H. (2015). Person tracking and following with 2d laser scanners. In Proceedings of the 2015 IEEE international conference on robotics and automation (ICRA) (pp. 726–733).Google Scholar
  26. Li, K., Wang, X., Xu, Y., & Wang, J. (2016). Density enhancement-based long-range pedestrian detection using 3-d range data. IEEE Transactions on Intelligent Transportation Systems, 17, 1368–1380.CrossRefGoogle Scholar
  27. Li, X. R., & Jilkov, V. P. (2003). Survey of maneuvering target tracking. part i: Dynamic models. IEEE Transactions on Aerospace and Electronic Systems, 39(4), 1333–1364.CrossRefGoogle Scholar
  28. Linder, T., Breuers, S., Leibe, B., & Arras, K. O. (2016). On multi-modal people tracking from mobile platforms in very crowded and dynamic environments. In Proceedings of the 2016 IEEE international conference on robotics and automation (ICRA) (pp. 5512–5519).Google Scholar
  29. Luber, M., & Arras, K. O. (2013). Multi-hypothesis social grouping and tracking for mobile robots. In Proceedings of Robotics: Science and Systems. Google Scholar
  30. Mohler, B. J., Thompson, W. B., Creem-Regehr, S. H., Pick, H. L., & Warren, W. H. (2007). Visual flow influences gait transition speed and preferred walking speed. Experimental Brain Research, 181(2), 221–228.CrossRefGoogle Scholar
  31. Moosmann, F., Pink, O., & Stiller, C. (2009). Segmentation of 3d lidar data in non-flat urban environments using a local convexity criterion. In Proceedings of the 2009 IEEE intelligent vehicles symposium (IV) (pp. 1931–0587).Google Scholar
  32. Munaro, M., & Menegatti, E. (2014). Fast RGB-D people tracking for service robots. Autonomous Robots, 37, 227–242.CrossRefGoogle Scholar
  33. Navarro-Serment, L. E., Mertz, C., & Hebert, M. (2009). Pedestrian detection and tracking using three-dimensional ladar data. In Proceedings of the 7th conference on field and service robotics (FSR) (pp. 103–112).Google Scholar
  34. Premebida, C., Carreira, J., Batista, J., & Nunes, U. (2014). Pedestrian detection combining RGB and dense LIDAR data. In Proceedings of the 2014 IEEE/RSJ international conference on intelligent robots and systems (IROS) (pp. 4112–4117).Google Scholar
  35. Quigley, M., Conley, K., Gerkey, B.P., Faust, J., Foote, T., Leibs, J., Wheeler, R., & Ng, A.Y. (2009). ROS: An open-source robot operating system. In DICRA workshop on open source software.Google Scholar
  36. Read, J., Bifet, A., Pfahringer, B., & Holmes, G. (2012). Batch-incremental versus instance-incremental learning in dynamic and evolving data. In Proceedings of the eleventh international symposium on intelligent data analysis (IDA 2012) (pp. 313–323).Google Scholar
  37. Rusu, R.B. (2009). Semantic 3D object maps for everyday manipulation in human living environments. Ph.D. thesis, Computer Science department, Technische Universitaet Muenchen, Germany.Google Scholar
  38. Rusu, R.B., & Cousins, S. (2011). 3D is here: Point Cloud Library (PCL). In Proceedings of the 2011 IEEE international conference on robotics and automation (ICRA).Google Scholar
  39. Shackleton, J., Voorst, B. V., & Hesch, J. A. (2010). Tracking people with a 360-degree lidar. In Proceedings of the Seventh IEEE international conference on advanced video and signal based surveillance (AVSS) (pp. 420–426).Google Scholar
  40. Spinello, L., & Arras, K.O. (2011). People detection in RGB-D data. In Proceedings of the 2011 IEEE/RSJ international conference on intelligent robots and systems (IROS) (pp. 3838–3843).Google Scholar
  41. Spinello, L., Luber, M., & Arras, K.O. (2011). Tracking people in 3d using a bottom-up top-down detector. In Proceedings of the 2011 IEEE international conference on robotics and automation (ICRA) (pp. 1304–1310).Google Scholar
  42. Sun, L., Yan, Z., Mellado, S.M., Hanheide, M., & Duckett, T. (2018). 3DOF pedestrian trajectory prediction learned from long-term autonomous mobile robot deployment data. In In proceedings of the 2018 IEEE international conference on robotics and automation (ICRA). Brisbane, Australia.Google Scholar
  43. Sun, L., Yan, Z., Zaganidis, A., Zhao, C., & Duckett, T. (2018). Recurrent-octomap: Learning state-based map refinement for long-term semantic mapping with 3-d-lidar data. IEEE Robotics and Automation Letters, 3(4), 3749–3756.CrossRefGoogle Scholar
  44. Teichman, A., Levinson, J., & Thrun, S. (2011). Towards 3d object recognition via classification of arbitrary object tracks. In Proceedings of the 2011 IEEE international conference on robotics and automation (ICRA) (pp. 4034–4041).Google Scholar
  45. Teichman, A., & Thrun, S. (2012). Tracking-based semi-supervised learning. International Journal of Robotics Research (IJRR), 31(7), 804–818.CrossRefGoogle Scholar
  46. Vintr, T., Yan, Z., Duckett, T., & Krajnik, T. (2019). Spatio-temporal representation for long-term anticipation of human presence in service robotics. In Proceedings of the 2018 IEEE international conference on robotics and automation (ICRA). Montreal, Canada.Google Scholar
  47. Wang, D. Z., & Posner, I. (2015). Voting for voting in online point cloud object detection. In Proceedings of Robotics: Science and Systems.Google Scholar
  48. Yan, Z., Duckett, T., & Bellotto, N. (2017). Online learning for human classification in 3D LiDAR-based tracking. In In Proceedings of the 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS). Vancouver, Canada.Google Scholar
  49. Yan, Z., Sun, L., Duckett, T., & Bellotto, N. (2018). Multisensor online transfer learning for 3d lidar-based human detection with a mobile robot. In Proceedings of the 2018 IEEE/RSJ international conference on intelligent robots and systems (IROS). Madrid, Spain.Google Scholar
  50. Zermas, D., Izzat, I., & Papanikolopoulos, N. (2017). Fast segmentation of 3d point clouds: A paradigm on lidar data for autonomous vehicle applications. In Proceedings of the 2018 IEEE international conference on robotics and automation (ICRA). Singapore.Google Scholar
  51. Zhou, Y., & Tuzel, O. (2018). Voxelnet: End-to-end learning for point cloud based 3d object detection. In Proceedings of the 2018 IEEE conference on computer vision and pattern recognition (CVPR) (pp. 4490–4499).Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Distributed Artificial Intelligence and Knowledge Laboratory (CIAD)University of Technology of Belfort-Montbéliard (UTBM)BelfortFrance
  2. 2.Lincoln Centre for Autonomous Systems (L-CAS)University of LincolnLincolnUK

Personalised recommendations