Skip to main content

Advertisement

SpringerLink
Log in

Knowledge-driven framework for industrial robotic systems

  • Published:
Journal of Intelligent Manufacturing Aims and scope Submit manuscript

Abstract

Due to their advantages, there is an increase of applying robotic systems for small batch production as well as for complex manufacturing processes. However, programming and configuring robots is time and resource consuming while being also accompanied by high costs that are especially challenging for small- and medium-sized enterprises. The current way of programming industrial robots by using teach-in control devices and/or using vendor-specific programming languages is in general a complex activity that requires extensive knowledge in the robotics domain. It is therefore important to offer new practical methods for the programming of industrial robots that provide flexibility and versatility in order to achieve feasible robotics solutions for small lot size productions. This paper focuses on the development of a knowledge-driven framework, which should overcome the limitations of state-of-the-art robotics solutions and enhance the agility and autonomy of industrial robotics systems using ontologies as a knowledge-source. The framework includes reasoning and perception abilities as well as the ability to generate plans, select appropriate actions, and finally execute these actions. In this context, a challenge is the fusion of vision system information with the decision-making component, which can use this information for generating the assembly tasks and executable programs. The introduced product model in the form of an ontology enables that the framework can semantically link perception data to product models to consequently derive handling operations and required tools. Besides, the framework enables an easier adaption of robot-based production systems for individualized production, which requires swift configuration and efficient planning. The presented approach is demonstrated in a laboratory environment with an industrial pilot test case. Our application shows the potential to reduce the efforts needed to program robots in an automated production environment. In this context, the benefits as well as shortcomings of the approach are also discussed in the paper.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Notes

  1. https://www.w3.org/TR/rdf-sparql-query/.

  2. Fast-Downward PDDL-Planner, available at http://www.fast-downward.org/.

  3. NodeJS, available at http://nodejs.org/.

  4. RDF/JS Data model specification, available at https://rdf.js.org/data-model-spec/.

  5. N3.js, available at https://github.com/rdfjs/N3.js/.

  6. SPARQL.js, available at https://github.com/RubenVerborgh/SPARQL.js/.

  7. Parsimmon, available at https://github.com/jneen/parsimmon.

  8. Handlebars.js, available at https://handlebarsjs.com/.

  9. Babylon.js, available at https://www.babylonjs.com/.

  10. Qt, available at https://www.qt.io/.

  11. OpenCV, available at https://opencv.org/.

  12. CUDA, available at https://developer.nvidia.com/cuda-downloads.

  13. StereoVision library, available at https://github.com/erget/StereoVision.

  14. VisPy, available at https://github.com/vispy/vispy.

  15. https://developer.microsoft.com/de-de/windows/kinect/.

  16. https://www.intelrealsense.com/depth-camera-d435/.

References

  • Al-Moadhen, A., Qiu, R., Packianather, M., Ji, Z., & Setchi, R.(2013). Integrating robot task planner with common-sense knowledge base to improve the efficiency of planning. Procedia Computer Science, 22:211 – 220. ISSN 1877-0509. https://doi.org/10.1016/j.procs.2013.09.097. 17th International conference in knowledge based and intelligent information and engineering systems-KES2013

  • Alterovitz, R., Koenig, S., & Likhachev, M. (2016). Robot planning in the real world: Research challenges and opportunities. AI Magazine, 37(2), 76–84. https://doi.org/10.1609/aimag.v37i2.2651.

    Article  Google Scholar 

  • Antoniou, G., & van Harmelen, F. (2004). Web ontology language: OWL. In S. Staab, & R. Studer (Eds.), Handbook on ontologies (pp. 67–92). Springer. ISBN 978-3-540-24750-0. https://doi.org/10.1007/978-3-540-24750-0

  • Bannat, A., Bautze, T., Beetz, M., Blume, J., Diepold, K., Ertelt, C., et al. (2011). Artificial cognition in production systems. IEEE Transactions on Automation Science and Engineering, 8(1), 148–174.

  • Binal, J., & Philipp, C. (2016). A knowledge-based architecture supporting declarative action representation for manipulation of everyday objects. In Proceedings of the 3rd workshop on model-driven robot software engineering, MORSE ’16 (pp. 40–46) Association for Computing Machinery. ISBN 9781450342599. https://doi.org/10.1145/3022099.3022105

  • Buoncompagni, L., Capitanelli, A., & Mastrogiovanni, F. (2017). A ros multi-ontology references services: Owl reasoners and application prototyping issues. arXiv preprint: arXiv:1706.10151

  • Carbonera, J., Fiorini, S., Prestes, E., Jorge, V., Abel, M., Madhavan, R., Locoro, A., Gonçalves, P., Haidegger, T. & Schlenoff, C. (2013). Defining position in a core ontology for robotics. In 2013 IEEE/RSJ international conference on intelligent robots and systems (pp. 1867–1872, 11). https://doi.org/10.1109/IROS.2013.6696603

  • Cashmore, M., Fox, M., Long, D., Magazzeni, D., Ridder, B., Carrera, A., Palomeras, N., Hurtos, N. & Carreras, M. (2015). Rosplan: Planning in the robot operating system. In Twenty-fifth international conference on automated planning and scheduling.

  • Chen, Y., & Medioni, G. (1992). Object modelling by registration of multiple range images. Image and Vision Computing, 10(3), 145–155. https://doi.org/10.1016/0262-8856(92)90066-C.

    Article  Google Scholar 

  • Crosby, M., Petrick, R. P. A., Toscano, C., Dias, R. C., Rovida, F., & Volker, Krüger. (2016). Integrating mission, logistics, and task planning for skills-based robot control in industrial kitting applications. In PlanSIG.

  • Diab, M., Akbari, A., Din, U., & Rosell, J. (2019). Pmk-a knowledge processing framework for autonomous robotics perception and manipulation. Sensors, 19, 03. https://doi.org/10.3390/s19051166.

    Article  Google Scholar 

  • Fox, M., & Long, D. (2003). Pddl2.1: An extension to pddl for expressing temporal planning domains. Journal of Artificial Intelligence Research, 20, 61–124. https://doi.org/10.1613/jair.1129.

    Article  Google Scholar 

  • Garg, S., Sünderhauf, N., Dayoub, F., Morrison, D., Cosgun, A., Carneiro, G., et al. (2020). Semantics for robotic mapping, perception and interaction: A survey. Foundations and Trends. Robotics, 8(1–2), 1–224. https://doi.org/10.1561/2300000059.

    Article  Google Scholar 

  • Ghadiri Nejad, M., Kovács, G., Vizvári, B., & Barenji, R. (2018). An optimization model for cyclic scheduling problem in flexible robotic cells. The International Journal of Advanced Manufacturing Technology, 95, 3863–3873. https://doi.org/10.1007/s00170-017-1470-z.

    Article  Google Scholar 

  • Ghadiri Nejad, M., Shavarani, M., Güden, H., & Barenji, R. (2019). Process sequencing for a pick-and-place robot in a real-life flexible robotic cell. The International Journal of Advanced Manufacturing Technology, 103, 1–15. https://doi.org/10.1007/s00170-019-03739-6.

    Article  Google Scholar 

  • Ghallab, M., Knoblock, C., Wilkins, D., Barrett, A., Christianson, D., Friedman, M., Kwok, C., Golden, K., Penberthy, S., Smith, D., Sun, Y., & Weld, D. (1998). Pddl-the planning domain definition language. Technical Report Tech Report CVC TR-98-003/DCS TR-1165, Yale Center for Computational Vision and Control, 08.

  • Gruber, T. R. (1995). Toward principles for the design of ontologies used for knowledge sharing? International Journal of Human-Computer Studies, 43(5), 907–928. https://doi.org/10.1006/ijhc.1995.1081.

    Article  Google Scholar 

  • Han, X.-F., Jin, J. S., Xie, J., Wang, M.-J., & Jiang, W. (2018). A comprehensive review of 3d point cloud descriptors. arXiv:1802.02297

  • Hoebert, T., Lepuschitz, W., & Merdan, M. (2019a). Automatic ontology-based plan generation for an industrial robotics system. In P. M. Roth, A. Pichler, R. Sablatnig, G. Stübl, & M. Vincze, (Eds.), Proceedings of the joint ARW and OAGM workshop 2019, Campus Steyr, Steyr, Austria, 2019 May 9–10. Austrian Robotics Workshop and Austrian Association for Pattern Recognition, University of Applied Sciences Upper Austria. ISBN 978-3-85125-663-5. https://doi.org/10.3217/978-3-85125-752-6-07

  • Hoebert, T., Lepuschitz, W, List, E., & Merdan, M. (2019b). Cloud-based digital twin for industrial robotics. In V. Mařík, P. Kadera, G. Rzevski, A. Zoitl, G. Anderst-Kotsis, A. Min Tjoa, & I. Khalil, (Eds.), Industrial applications of holonic and multi-agent systems (pp. 105–116). Springer. ISBN 978-3-030-27878-6.

  • Huckaby, J. O., & Christensen, H. I. (2013). Toward a knowledge transfer framework for process abstraction in manufacturing robotics. In Proceedings of the 30th international conference on machine learning, Atlanta, Georgia (Vol. 28).

  • Iriondo, A., Lazkano, E., Susperregi, L., Urain, J., Fernandez, A., & Molina, J. (2019). Pick and place operations in logistics using a mobile manipulator controlled with deep reinforcement learning. Applied Sciences, 9(2), 348. https://doi.org/10.3390/app9020348.

    Article  Google Scholar 

  • Jacobsson, L., Malec, J., & Nilsson, K. (2016). Modularization of skill ontologies for industrial robots. In Proceedings of ISR 2016: 47st international symposium on robotics (pp. 1–6).

  • Jiang, Y., Zhang, S., Khandelwal, P., & Stone, P. (2018). Task planning in robotics: An empirical comparison of pddl-based and asp-based systems.

  • Kast, B., Albrecht, S., Feiten, W., & Zhang, J. (2019). Bridging the gap between semantics and control for industry 4.0 and autonomous production. In 2019 IEEE 15th international conference on automation science and engineering (CASE) (pp. 780–787). https://doi.org/10.1109/COASE.2019.8843174

  • Kattepur, A., & Purushotaman, B. (2020). Roboplanner: A pragmatic task planning framework for autonomous robots. Cognitive Computation and Systems, 2:12–22(10).

  • Kharlamov, E., Grau, B. C., Jiménez-Ruiz, E., Lamparter, S., Mehdi, G., Ringsquandl, M., Nenov, Y., Grimm, S., Roshchin, M., & Horrocks, I. (2016). Capturing industrial information models with ontologies and constraints. In P. Groth, E. Simperl, A. Gray, M. Sabou, M. Krötzsch, F. Lecue, F. Flöck, & Y. Gil, (Eds.), The semantic web–ISWC 2016 (pp. 325–343). Springer. ISBN 978-3-319-46547-0

  • Kootbally, Z., Kramer, T. R., Schlenoff, C., & Gupta, S. K. (2018). Implementation of an ontology-based approach to enable agility in kit building applications. International Journal of Semantic Computing, 12(01), 5–24. https://doi.org/10.1142/S1793351X18400019.

    Article  Google Scholar 

  • Kootbally, Z., Schlenoff, C., Lawler, C., Kramer, T., & Gupta, S. K. (2015). Towards robust assembly with knowledge representation for the planning domain definition language (pddl). Robotics and Computer-Integrated Manufacturing, 33, 42–55. https://doi.org/10.1016/j.rcim.2014.08.006. Special Issue on Knowledge Driven Robotics and Manufacturing

  • Krötzsch, M., Simancik, F., & Horrocks, I. (2014). Description logics. IEEE Intelligent Systems, 29(1), 12–19.

    Article  Google Scholar 

  • Lepuschitz, W., Zoitl, A., Vallée, M., & Merdan, M. (2011). Toward self-reconfiguration of manufacturing systems using automation agents. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 41(1), 52–69.

    Article  Google Scholar 

  • Löfving, M., Almström, P., Jarebrant, C., Wadman, B., & Widfeldt, M. (2018). Evaluation of flexible automation for small batch production. Procedia Manufacturing, 25, 177–184. https://doi.org/10.1016/j.promfg.2018.06.072.

    Article  Google Scholar 

  • Lohmann, S., Link, V., Marbach, E., & Negru, S. (2015). WebVOWL: Web-based visualization of ontologies. In Proceedings of EKAW 2014 satellite events, volume 8982 of LNAI (pp. 154–158). Springer.

  • Manzoor, S., Rocha, Y. G., Joo, S.-H., Bae, S.-H., Kim, E.-J., Joo, K.-J., et al. (2021). Ontology-based knowledge representation in robotic systems: A survey oriented toward applications. Applied Sciences,. https://doi.org/10.3390/app11104324.

    Article  Google Scholar 

  • Merdan, M., Hoebert, T., List, E., & Lepuschitz, W. (2019). Knowledge-based cyber-physical systems for assembly automation. Production and Manufacturing Research, 7(1), 223–254.

    Article  Google Scholar 

  • Olivares-Alarcos, A., Beßler, D., Khamis, A., Goncalves, P., Habib, M. K., Bermejo-Alonso, J., et al. (2019). A review and comparison of ontology-based approaches to robot autonomy. The Knowledge Engineering Review, 34, e29. https://doi.org/10.1017/S0269888919000237.

  • Parente, M., Figueira, G., Amorim, P., & Marques, A. (2020). Production scheduling in the context of industry 4.0: review and trends. International Journal of Production Research, 58 (17):5401–5431, https://doi.org/10.1080/00207543.2020.1718794.

  • Pedersen, M. R., & Krüger, V. (2015). Automated planning of industrial logistics on a skill-equipped robot. In Workshop on task planning for intelligent robots in service and manufacturing, IEEE international conference on intelligent robots and systems. proceedings. IROS Hamburg. null ; Conference date: 02-10-2015 Through 02-10-2015.

  • Perzylo, A., Rickert, M., Kahl, B., Somani, N., Lehmann, C., Kuss, A., et al. (2019). Smerobotics: Smart robots for flexible manufacturing. IEEE Robotics Automation Magazine, 26(1), 78–90.

    Article  Google Scholar 

  • Perzylo, A., Somani, N., Rickert, M., & Knoll, A. (2015). An ontology for cad data and geometric constraints as a link between product models and semantic robot task descriptions. In 2015 IEEE/RSJ international conference on intelligent robots and systems (IROS), (pp. 4197–4203). https://doi.org/10.1109/IROS.2015.7353971.

  • Proctor, F., Balakirsky, S., Kootbally, Z., Kramer, T., Schlenoff, C., & Shackleford, W. (2016). The canonical robot command language (crcl). Industrial Robot: An International Journal, 43, 495–502. https://doi.org/10.1108/IR-01-2016-0037.

    Article  Google Scholar 

  • Rogalla, A., Fay, A. & Niggemann, O. (2018). Improved domain modeling for realistic automated planning and scheduling in discrete manufacturing. In 2018 IEEE 23rd international conference on emerging technologies and factory automation (ETFA) (Vol. 1, pp. 464–471).

  • Rovida, F., Crosby, M., Holz, D., Polydoros, A. S., Großmann, B., Petrick, R. P. A., et al. (2017). Skiros–a skill-based robot control platform on top of ros. In A. Koubaa (Ed.), Robot operating system (ROS): The complete reference (Vol. 2, pp. 121–160). Springer.

  • Rusu, R. B., & Cousins, S. (2011). 3d is here: Point cloud library (pcl). In 2011 IEEE International conference on robotics and automation (pp. 1–4). https://doi.org/10.1109/ICRA.2011.5980567

  • Rusu, R. B., Blodow, N., & Beetz, M. (2009). Fast point feature histograms (fpfh) for 3d registration. In Proceedings of the 2009 IEEE international conference on robotics and automation, ICRA’09 (pp. 1848–1853). IEEE Press. ISBN 9781424427888

  • R. B. Rusu, G. Bradski, R. Thibaux, & J. Hsu. (2010). Fast 3d recognition and pose using the viewpoint feature histogram. In 2010 IEEE/RSJ international conference on intelligent robots and systems (pp. 2155–2162).

  • Sampath Kumar, V. R., Khamis, A., Fiorini, S., Carbonera, J. L., Alarcos, A. O., Habib, M., et al. (2019). Ontologies for industry 4.0. The Knowledge Engineering Review, 34(e17), https://doi.org/10.1017/S0269888919000109.

  • Saravanan, S., Ramanathan, K. C., Ramya, M. M., & Janardhanan, M. N. (2020). Review on state-of-the-art dynamic task allocation strategies for multiple-robot systems. Industrial Robot,. https://doi.org/10.1108/IR-04-2020-0073.

    Article  Google Scholar 

  • Schlenoff, C., Prestes, E., Madhavan, R., Goncalves, P., Li, H., Balakirsky, S., Kramer, T., & Migueláñez, E. (2012). An ieee standard ontology for robotics and automation. In 2012 IEEE/RSJ international conference on intelligent robots and systems (pp. 1337–1342).

  • Silva do Monte Lima, J. P. & Teichrieb V. (2016). An efficient global point cloud descriptor for object recognition and pose estimation. In 2016 29th SIBGRAPI conference on graphics, patterns and images (SIBGRAPI) (pp. 56-63).

  • Stenmark, M., & Malec, J. (2015). Knowledge-based instruction of manipulation tasks for industrial robotics. Robotics and Computer-Integrated Manufacturing, 33, 56–67. https://doi.org/10.1016/j.rcim.2014.07.004.

    Article  Google Scholar 

  • Sun, X., & Zhang, Y. (2019). A review of domain knowledge representation for robot task planning. In Proceedings of the 2019 4th international conference on mathematics and artificial intelligence, ICMAI 2019 (pp. 176–183). Association for Computing Machinery. ISBN 9781450362580. https://doi.org/10.1145/3325730.3325756.

  • Szeliski, R. (2010). Computer vision: Algorithms and applications. London: Texts in Computer Science. Springer. ISBN 9781848829350

  • Tenorth, M., & Beetz, M. (2013). Knowrob: A knowledge processing infrastructure for cognition-enabled robots. The International Journal of Robotics Research, 32(5), 566–590. https://doi.org/10.1177/0278364913481635.

    Article  Google Scholar 

  • Towards a Robot Task Ontology Standard, volume Volume 3: Manufacturing Equipment and Systems of international manufacturing science and engineering conference

  • Wally, B., Vyskočil, J., Novak, P., Huemer, C., Sindelar, R., Kadera, P., Mazak, A., & Wimmer, M. (2019). Production planning with iec 62264 and pddl. In 2019 IEEE 17th international conference on industrial informatics (INDIN) (Vol, 1, pp. 492–499). https://doi.org/10.1109/INDIN41052.2019.8972050

  • Wang, S., Zhang, Y., & Liao, Z. (2019a). Review on the knowledge graph in robotics domain. In 3rd International conference on computer engineering, information science and application technology (ICCIA 2019) (pp. 424–431). Atlantis Press, 2019/07. ISBN 978-94-6252-760-7. https://doi.org/10.2991/iccia-19.2019.65

  • Wang, F., Liang, C., Ru, C., & Cheng, H. (2019b). An improved point cloud descriptor for vision based robotic grasping system. Sensors, 19, 2225. https://doi.org/10.3390/s19102225.

    Article  Google Scholar 

  • Zander, S., Ahmed, N., & Frank, M. T. (2016). A survey about the usage of semantic technologies for the description of robotic components and capabilities. In SAMI@iKNOW.

Download references

Acknowledgements

The authors acknowledge the financial support from the “Production of the Future” program of the Austrian Ministry for Climate Action, Environment, Energy, Mobility, Innovation and Technology under contract FFG 858707 as well as from the “netidee” program of the Internet Foundation Austria under contract no. 4647. This work has its roots in the Master Thesis of Timon Hoebert.

Author information

Authors and Affiliations

  1. Practical Robotics Institute Austria, Vienna, Austria

    Timon Hoebert, Wilfried Lepuschitz & Munir Merdan

  2. Vision4Robotics Group, Automation and Control Institute, Vienna University of Technology, Vienna, Austria

    Markus Vincze

Authors
  1. Timon Hoebert
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wilfried Lepuschitz
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Markus Vincze
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Munir Merdan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Munir Merdan.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hoebert, T., Lepuschitz, W., Vincze, M. et al. Knowledge-driven framework for industrial robotic systems. J Intell Manuf 34, 771–788 (2023). https://doi.org/10.1007/s10845-021-01826-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10845-021-01826-8

Keywords

Search

Navigation