Skip to main content
SpringerLink
Log in

Item Ownership Relationship Semantic Learning Strategy for Personalized Service Robot

  • Research Article
  • Published:
International Journal of Automation and Computing Aims and scope Submit manuscript

Abstract

In order to satisfy the robotic personalized service requirements that can select exclusive items to perform inference and planning according to different service individuals, the service robots need to have the ability to independently obtain the ownership relationship between humans and their carrying items. In this work, we present a novel semantic learning strategy for item ownership. Firstly, a human-carrying-items detection network based on human posture estimation and object detection model is used. Then, the transferred convolutional neural network is used to extract the characteristics of the objects and the back-end classifier to recognize the object instance. At the same time, the face detection and recognition model are used to identify the service individual. Finally, on the basis of the former two, the active learning of ownership items is completed. The experimental results show that the proposed ownership semantic learning strategy can determine the ownership relationship of private goods accurately and efficiently. The solution of this problem can improve the intelligence level of robot life service.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Similar content being viewed by others

References

  1. H. Wu, X. J. Wu, Q. Ma, G. H. Tian. Cloud robot: Semantic map building for intelligent service task. Applied Intelligence, vol.49, no. 2, pp. 319–334, 2019. DOI: https://doi.org/10.1007/s10489-018-1277-0.

    Article  Google Scholar 

  2. W. He, Z. J. Li, C. L. P. Chen. A survey of Human-centered Intelligent robots: Issues and challenges. IEEE/CAA Journal of Automatica Sinica, vol.4, no.4, pp. 602–609, 2017. DOI: https://doi.org/10.1109/JAS.2017.7510604.

    Google Scholar 

  3. Y. Yang, F. Qiu, H. Li, L. Zhang, M. L. Wang, M. Y. Fu. Large-scale 3D semantic mapping using stereo vision. International Journal of Automation and Computing, vol.15, no. 2, pp. 194–206, 2018. DOI: https://doi.org/10.1007/s11633-018-1118-y.

    Article  Google Scholar 

  4. E. Daglarh, S. F. Daglarh, G. O. Giinel, H. Kose. Improving human-robot interaction based on joint attention. Applied Intelligence, vol.47, no. 1, pp. 62–82, 2017. DOI: https://doi.org/10.1007/sl0489-016-0876-x.

    Article  Google Scholar 

  5. T. M. Wang, Y. Tao, H. Liu. Current researches and future development trend of intelligent robot: A review. International Journal of Automation and Computing, vol.15, no. 5, pp. 525–546, 2018. DOI: https://doi.org/10.1007/s11633-018-1115-1.

    Article  Google Scholar 

  6. J. Rivera-Rubio, S. Idrees, I. Alexiou, L. Hadjilucas, A. A. Bharath. Small hand-held object recognition test (SHORT). In Proceedings of IEEE Winter Conference on Applications of Computer Vision, IEEE, Steamboat Springs, USA, pp. 524–531, 2014. DOI: https://doi.org/10.1109/WACV.2014.6836057.

    Chapter  Google Scholar 

  7. J. Rivera-Rubio, S. Idrees, I. Alexiou, L. Hadjilucas, A. A. Bharath. A dataset for hand-held object recognition. In Proceedings of IEEE International Conference on Image Processing, IEEE, Paris, France, pp. 5881–5885, 2014. DOI: https://doi.org/10.1109/ICIP.2014.7026188.

    Google Scholar 

  8. J. W. Hsieh, J. C. Cheng, L. C. Chen, C. H. Chuang, D. Y. Chen. Handheld object detection and its related event analysis using ratio histogram and mixture of HMMs. Journal of Visual Communication and Image Representation, vol. 25, no. 6, pp. 1399–1415, 2014. DOI: https://doi.org/10.1016/j.jvcir.2014.05.009.

    Article  Google Scholar 

  9. X. Lv, S. Q. Jiang, L. Herranz, S. Wang. RGB-D handheld object recognition based on heterogeneous feature fusion. Journal of Computer Science and Technology, vol. 30, no.2, pp.340–352, 2015. DOI: https://doi.org/10.1007/s11390-015-1527-0.

    Article  Google Scholar 

  10. X. Li, S. Q. Jiang, X. Lv, C. P. Chen. Learning to recognize hand-held objects from scratch. In Proceedings of the 17th Pacific-Rim Conference on Multimedia, Springer, Xi’an, China, pp. 527–539, 2016. DOI: https://doi.org/10.1007/978-3-319-48896-7_52.

    Google Scholar 

  11. K. Yamaguchi, M. Hadi Kiapour, L. E. Ortiz, T. L. Berg. Parsing clothing in fashion photographs. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Providence, USA, pp. 3570–3577, 2012. DOI: https://doi.org/10.1109/CVPR.2012.6248101.

    Google Scholar 

  12. X. D. Liang, C. Y. Xu, X. H. Shen, J. C. Yang, S. Liu, J. H. Tang, L. Lin, S. C. Yan. Human parsing with contextualized convolutional neural network. IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.39, no. 1, pp. 115–127, 2017. DOI: https://doi.org/10.1109/TPAMI.2016.2537339.

    Article  Google Scholar 

  13. K. Gong, X. D. Liang, D. Y. Zhang, X. H. Shen, L. Lin. Look into person: Self-supervised structure-sensitive learning and a new benchmark for human parsing. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Honolulu, USA, pp. 6757–6765, 2017. DOI: https://doi.org/10.1109/CVPR.2017.715.

    Google Scholar 

  14. X. J. Chen, R. Mottaghi, X. B. Liu, S. Fidler, R. Urtasun, A. Yuille. Detect what you can: Detecting and representing objects using holistic models and body parts. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Columbus, USA, pp. 1979–1986, 2014. DOI: https://doi.org/10.1109/CVPR.2014.254.

    Google Scholar 

  15. J. S. Li, J. Zhao, Y. C. Wei, C. Y. Lang, Y. D. Li, T. Sim, S. C. Yan, J. S. Feng. Multiple-human parsing in the wild, Available: https://arxiv.org/abs/1705.07206, March 2018.

    Google Scholar 

  16. Z. Cao, T. Simon, S. E. Wei, Y. Sheikh. Realtime multi-person 2D pose estimation using part affinity fields. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Honolulu, USA, pp. 1302–1310, 2017. DOI: https://doi.org/10.1109/CVPR.2017.143.

    Google Scholar 

  17. W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C. Y. Fu, A. C. Berg. SSD: Single shot MultiBox detector. In Proceedings of the 14th European Conference on Computer Vision, Springer, Amsterdam, The Netherlands, pp. 21–37, 2016. DOI: https://doi.org/10.1007/978-3-319-46448-02.

    Google Scholar 

  18. P. Felzenszwalb, D. McAllester, D. Ramanan. A discrim-inatively trained, multiscale, deformable part model. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Anchorage, USA, 2008. DOI: https://doi.org/10.1109/CVPR.2008.4587597.

    Google Scholar 

  19. R. Girshick, J. Donahue, T. Darrell, J. Malik. Rich feature hierarchies for accurate object detection and semantic seg-mentation. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Columbus, USA, pp. 580–587, 2014. DOI: https://doi.org/10.1109/CVPR.2014.81.

    Google Scholar 

  20. R. Girshick. Fast R-CNN. In Proceedings of 2015 IEEE International Conference on Computer Vision, IEEE, Santiago, Chile, pp. 1440–1448, 2015. DOI: https://doi.org/10.1109/ICCV.2015.169.

    Google Scholar 

  21. S. Q. Ren, K. M. He, R. Girshick, J. Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In Proceedings of the 28th International Conference on Neural Information Processing Systems, MIT Press, Montreal, Canada, pp. 91–99, 2015.

    Google Scholar 

  22. J. Redmon, S. Divvala, R. Girshick, A. Farhadi. You only look once: Unified, real-time object detection. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Las Vegas, USA, pp. 779–788, 2016. DOI: https://doi.org/10.1109/CVPR.2016.91.

    Google Scholar 

  23. K. Simonyan, A. Zisserman. Very deep convolutional net-works for large-scale image recognition, Available: https://arxiv.org/abs/1409.1556, April 2015.

    Google Scholar 

  24. T. Y. Lin, P. Goyal, R. Girshick, K. M. He, P. Dollar. Focal loss for dense object detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, published on-line. DOI: https://doi.org/10.1109/TPAMI.2018.2858826.

    Article  Google Scholar 

  25. L. Pishchulin, A. Jain, M. Andriluka, T. Thormahlen, B. Schiele. Articulated people detection and pose estimation: Reshaping the future. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Providence, USA, pp. 3178–3185, 2012. DOI: https://doi.org/10.1109/CV-PR.2012.6248052.

    Google Scholar 

  26. G. Gkioxari, B. Hariharan, R. Girshick, J. Malik. Using k-poselets for detecting people and localizing their keypoints. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Columbus, USA, pp. 3582–3589, 2014. DOI: https://doi.org/10.1109/CVPR.2014.458.

    Google Scholar 

  27. L. Pishchulin, E. Insafutdinov, S. Y. Tang, B. Andres, M. Andriluka, P. Gehler, B. Schiele. DeepCut: Joint subset partition and labeling for multi person pose estimation. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Las Vegas, USA, pp.4929–1937, 2016. DOI: https://doi.org/10.1109/CVPR.2016.533.

    Google Scholar 

  28. E. Insafutdinov, L. Pishchulin, B. Andres, M. Andriluka, B. Schiele. Deepercut: A deeper, stronger, and faster multi-person pose estimation model. In Proceedings of the 14th European Conference on Computer Vision, Springer, Amsterdam, The Netherlands, pp. 34–50, 2016. DOI: https://doi.org/10.1007/978-3-319-46466-43.

    Google Scholar 

  29. S. Yang, P. Luo, C. C. Loy, X. O. Tang. WIDER FACE: A face detection benchmark. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Las Vegas, USA, pp. 5525–5533, 2016. DOI: https://doi.org/10.1109/CV-PR.2016.596.

    Google Scholar 

  30. Z. Liu, P. Luo, X. G. Wang, X. O. Tang. Deep learning face attributes in the wild. In Proceedings of IEEE Inter-national Conference on Computer Vision, IEEE, Santiago, Chile, pp. 3730–3738, 2015. DOI: https://doi.org/10.1109/ICCV.2015.425.

    Google Scholar 

Download references

Acknowledgements

This work was supported by the Joint Funds of National Natural Science Foundation of China (Nos. U1813215 and 2018YFB1307101), National Natural Science Foundation of China (Nos. 61603213, 61773239, 61973187, 61973192 and 91748115), Shandong Provincial Natural Science Foundation, China (No.ZR2017MF014), Jinan Technology project (No. 20150219) and Taishan Scholars Programme of Shandong Province.

Author information

Authors and Affiliations

  1. School of Control Science and Engineering, Shandong University, Jinan, 250061, China

    Hao Wu, Zhao-Wei Chen, Guo-Hui Tian, Qing Ma & Meng-Lin Jiao

Authors
  1. Hao Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhao-Wei Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Guo-Hui Tian
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Qing Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Meng-Lin Jiao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Guo-Hui Tian.

Additional information

Hao Wu received the M.Sc. and Ph.D. degrees in control theory and control engineering from Shandong University, China in 1997 and 2011, respectively. Currently, she is an associate professor at Shandong University, China. Her research mainly focuses on the navigation of service robot

Her research interests include robotics, control theory and automatic control system.

Zhao-Wei Chen received the B.Sc. degree in automation from Shandong University, China in 2017. He is currently a master student at Shandong University, China.

His research interests include deep learning and computer vision.

Guo-Hui Tian received the M.Sc. degree in industry automation from Shandong University, China in 1993, and received the Ph. D. degree in automatic control theory and application from Northeast University, China in 1997. He has a postdoctoral research at Engineering Department of Tokyo University, Japan from 2003 to 2005. Currently, he is a professor and a doctoral tutor at Shandong University, China. His research mainly focuses on service robot and smart space.

His research interests include robotics, feedback control systems, and control theory.

Qing Ma received the M.Sc. and Ph.D. degrees in control theory and control engineering from Shandong University, China in 2002 and 2012, respectively. Currently, he is a lecturer at Shandong University, China.

His research interests include machine learning and control theory.

Meng-Lin Jiao received the B.Sc. degree in automation from Shandong University, China in 2018. He is a master student in control science and engineering at Shandong University, China.

His research interests include machine learning and navigation of service robot.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wu, H., Chen, ZW., Tian, GH. et al. Item Ownership Relationship Semantic Learning Strategy for Personalized Service Robot. Int. J. Autom. Comput. 17, 390–402 (2020). https://doi.org/10.1007/s11633-019-1206-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11633-019-1206-7

Keywords

Search

Navigation