Skip to main content
SpringerLink
Log in

Semantic Temporal Object Search System Based on Heat Maps

  • Regular paper
  • Published:
Journal of Intelligent & Robotic Systems Aims and scope Submit manuscript

Abstract

Service robots that operate in human-populated environments are exposed to changes in the environment. Most mobile robot are concerned with portable objects only if there is a risk of colliding with them. However, humans need to keep track of especially personal items - like a cup, glasses, the mobile phone, etc - that can be left at various locations in the living environment. Thus, we think it will be useful if a robot assistant can support a human in keeping track of such item. In this paper, we therefore study the problem of object search (OS) in unknown indoor environments. We present an OS system that relies on semantic information inferred from the changes in the objects’ position over time in the environment, which allows the robot to reduce search costs by giving preference to more promising regions for the target object. Our two-mode OS system gathers data from the objects’ placement by executing its recording mode, which is later used when the robot executes the request mode to search for the target object. We compared the performance of our semantic, temporal OS system with two other search methods in simulation, considering six different scenarios with objects being moved from time to time. Moreover, we also demonstrate our OS system’s efficiency in the HH106 dataset collected over two months containing person occurrences in a residential environment. The experiments in simulation indicate that our semantic, temporal OS system always finds the target object and, in some cases, with the robot travelling a distance two times smaller. The tests with the HH106 enhance our method’s robustness and efficiency in properly estimating the person’s position at different times of the day.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Portugal, D., Araújo, A.G., Couceiro, M.S.: Improving the robustness of a service robot for continuous indoor monitoring: An incremental approach. Int. J. Adv. Robot. 18(3), (2021)

  2. Lu, V.N., Wirtz, J., Kunz, W.H., Paluch, S., Gruber, T., Martins, A., Patterson, P.G.: Service robots, customers and service employees: what can we learn from the academic literature and where are the gaps? Journal of Service Theory and Practice (2020)

  3. Paulius, D., Sun, Y.: A survey of knowledge representation in service robotics. Rob. Auton. Syst. 118, 13–30 (2019)

    Article  Google Scholar 

  4. Torresen, J., Kurazume, R., Prestes, E.: Special issue on elderly care robotics - technology and ethics. J. Intell. Robot. Syst. 98(1), 3–4 (2020)

    Article  Google Scholar 

  5. Torresen, J., Schulz, T., Uddin, Z., Khaksar, W., Prestes, E.: Robot companions for older people – ethical concerns. In: International Conference on Robot Ethics and Standards, 53 (2018)

  6. Sprute, D., Pörtner, A., Rasch, R., Battermann, S., König, M.: Ambient assisted robot object search, pp. 112–123. Springer (2017)

  7. Liu, S., Tian, G., Zhang, Y., Zhang, M., Liu, S.: Service planning oriented efficient object search: A knowledge-based framework for home service robot. Expert Syst. Appl. 187, 115853 (2022)

    Article  Google Scholar 

  8. Zhang, Y., Tian, G., Shao, X., Zhang, M., Liu, S.: Semantic grounding for long-term autonomy of mobile robots towards dynamic object search in home environments. IEEE Transactions on Industrial Electronics (2022)

  9. Rasouli, A., Lanillos, P., Cheng, G., Tsotsos, J.K.: Attention-based active visual search for mobile robots. Auton. Robots 44(2), 131–146 (2020)

    Article  Google Scholar 

  10. Cabanillas, J., Morales, E.F., Sucar, L.E.: An efficient strategy for fast object search considering the robot’s perceptual limitations, pp. 552–561. Springer (2010)

  11. Deyle, T., Reynolds, M.S., Kemp, C.C.: Finding and navigating to household objects with uhf rfid tags by optimizing rf signal strength, pp. 2579–2586 (2014)

  12. Chen, X., Lee, S.: Visual search of an object in cluttered environments for robotic errand service, pp. 4060–4065 (2013)

  13. Meger, D., Muja, M., Helmer, S., Gupta, A., Gamroth, C., Hoffman, T., Baumann, M., Southey, T., Fazli, P., Wohlkinger, W., et al.: Curious george: An integrated visual search platform, pp. 107–114. IEEE (2010)

  14. Forssen, P.-E., Meger, D., Lai, K., Helmer, S., Little, J.J., Lowe, D.G.: Informed visual search: Combining attention and object recognition, pp. 935–942 (2008)

  15. Zhou, T., Wang, W., Qi, S., Ling, H., Shen, J.: Cascaded human-object interaction recognition, pp. 4263–4272 (2020)

  16. Krajník, T., Vintr, T., Broughton, G., Majer, F., Rouček, T., Ulrich, J., Blaha, J., Pěčonková, V., Rektoris, M.: Chronorobotics: Representing the structure of time for service robots (2020)

  17. Aydemir, A., Pronobis, A., Göbelbecker, M., Jensfelt, P.: Active visual object search in unknown environments using uncertain semantics. IEEE Trans. Robot. 29(4), 986–1002 (2013)

    Article  Google Scholar 

  18. Wang, C., Cheng, J., Wang, J., Li, X., Meng, M.Q.-H.: Efficient object search with belief road map using mobile robot. IEEE Robot. Autom. Lett. 3(4), 3081–3088 (2018)

    Article  Google Scholar 

  19. Ye, Y., Tsotsos, J.K.: Sensor planning for 3d object search. Comput. Vis. Image Underst. 73(2), 145–168 (1999)

    Article  Google Scholar 

  20. Aydemir, A., Pronobis, A., Sjöö, K., Göbelbecker, M., Jensfelt, P.: Object search guided by semantic spatial knowledge. The RSS 11 (2011)

  21. Aydemir, A., Göbelbecker, M., Pronobis, A., Sjöö, K., Jensfelt, P.: Plan-based object search and exploration using semantic spatial knowledge in the real world., pp. 13–18 (2011)

  22. Aydemir, A., Sjöö, K., Folkesson, J., Pronobis, A., Jensfelt, P.: Search in the real world: Active visual object search based on spatial relations, pp. 2818–2824. IEEE (2011)

  23. Aydemir, A., Jensfelt, P.: Exploiting and modeling local 3d structure for predicting object locations, pp. 3885–3892. IEEE (2012)

  24. Oksanen, J., Bergman, C., Sainio, J., Westerholm, J.: Methods for deriving and calibrating privacy-preserving heat maps from mobile sports tracking application data. J. Transp. Geogr. 48, 135–144 (2015)

    Article  Google Scholar 

  25. Krajník, T., Fentanes, J.P., Mozos, O.M., Duckett, T., Ekekrantz, J., Hanheide, M.: Long-term topological localisation for service robots in dynamic environments using spectral maps, pp. 4537–4542. IEEE (2014)

  26. Vintr, T., Yan, Z., Duckett, T., Krajník, T.: Spatio-temporal representation for long-term anticipation of human presence in service robotics, pp. 2620–2626. IEEE (2019)

  27. Narayana, M., Kolling, A., Nardelli, L., Fong, P.: Lifelong update of semantic maps in dynamic environments. In: 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 6164–6171. IEEE (2020)

  28. Krajník, T., Kulich, M., Mudrová, L., Ambrus, R., Duckett, T.: Where’s waldo at time t? using spatio-temporal models for mobile robot search, pp. 2140–2146. IEEE (2015)

  29. Bjelonic, M.: YOLO ROS: Real-Time Object Detection for ROS. (2016–2018). https://github.com/leggedrobotics/darknet_ros. Accessed February 2020

  30. Gerkey, B.: Gmapping package: laser-based Simultaneous Localization and Mapping for ROS. (2013–2020). http://wiki.ros.org/gmapping. Accessed February 2020

  31. Redmon, J., Farhadi, A.: Yolov3: An incremental improvement. (2018) arXiv:1804.02767

  32. Vittorio, A.: Toolkit to download and visualize single or multiple classes from the huge Open Images v4 dataset. Github (2018)

  33. Kinney, C.: HEATMAPS, United States Patent and Trademark Office, United States, pp. 75263259 (1993)

  34. Afif, M., Ayachi, R., Said, Y., Pissaloux, E., Atri, M.: A novel dataset for intelligent indoor object detection systems. Artificial Intelligence Advances, 1 (2019). https://doi.org/10.30564/aia.v1i1.925

  35. Cook, D.J.: Learning setting-generalized activity models for smart spaces. IEEE Intell. Syst. 2010(99), 1 (2010)

    Google Scholar 

  36. Zhang, L., Wei, L., Shen, P., Wei, W., Zhu, G., Song, J.: Semantic slam based on object detection and improved octomap. IEEE Access 6, 75545–75559 (2018)

    Article  Google Scholar 

Download references

Funding

The TITAN Xp used for this research was donated by the NVIDIA Corporation. This study was financed in part by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - Brazil (CAPES) - Finance Code 001, CNPq. Besides, this work is also partially supported by the Research Council of Norway (RCN) as a part of the COINMAC projects (grant agreement 261645 and 309869), the PIRC project (grant agreement 312333) and the VIROS project (grant agreement 288285).

Author information

Author notes

    Authors and Affiliations

    1. Institute of Informatics, Federal University of Rio Grande do Sul, Rio Grande do Sul, Brazil

      Mathias Mantelli, Diego Pittol, Renan Maffei & Mariana Kolberg

    2. Department of Informatics, University of Oslo, Oslo, Norway

      Farzan M. Noori & Jim Torresen

    Authors
    1. Mathias Mantelli
      View author publications

      You can also search for this author in PubMed Google Scholar

    2. Farzan M. Noori
      View author publications

      You can also search for this author in PubMed Google Scholar

    3. Diego Pittol
      View author publications

      You can also search for this author in PubMed Google Scholar

    4. Renan Maffei
      View author publications

      You can also search for this author in PubMed Google Scholar

    5. Jim Torresen
      View author publications

      You can also search for this author in PubMed Google Scholar

    6. Mariana Kolberg
      View author publications

      You can also search for this author in PubMed Google Scholar

    Contributions

    MM, DP, RM, and MK conceived and designed the approach. FN and JT worked with the object detection module. MM carried out the experiments, and the data analysis was performed by MM, DP, RM and MK. All authors wrote the manuscript and reviewed its final version.

    Corresponding author

    Correspondence to Mathias Mantelli.

    Ethics declarations

    Conflicts of interest

    The authors declare there is no conflict of interest in this paper. The code, data and any other document will be release in the repository of the Phi Robotics Research Lab, as soon as all the files meet the Google Style Guide and are well documented.

    Ethics approval

    Not applicable.

    Consent to participate

    Not applicable.

    Consent to publish

    Not applicable.

    Additional information

    Publisher’s Note

    Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

    Mathias Fassini Mantelli, Farzan M. Noori, Diego Pittol, Renan de Queiroz Maffei, Jim Torresen and Mariana Luderitz Kolberg are contributed equally to this work.

    Rights and permissions

    Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

    Reprints and permissions

    About this article

    Check for updates. Verify currency and authenticity via CrossMark

    Cite this article

    Mantelli, M., Noori, F.M., Pittol, D. et al. Semantic Temporal Object Search System Based on Heat Maps. J Intell Robot Syst 106, 69 (2022). https://doi.org/10.1007/s10846-022-01760-8

    Download citation

    • Received:

    • Accepted:

    • Published:

    • DOI: https://doi.org/10.1007/s10846-022-01760-8

    Keywords

    Search

    Navigation