Journal of Intelligent & Robotic Systems

, Volume 96, Issue 3–4, pp 439–456 | Cite as

Quantitative Assessment at Task-Level for Performance of Robotic Configurations and Task Plans

  • Ching-Yen WengEmail author
  • Wei Chian Tan
  • Qilong Yuan
  • I-Ming Chen


Given a robotic manipulation task, decision on which robotic configuration (robotic system and necessary peripherals, including assistant tools, sensor systems, and so on.) to use and evaluating performance of the solution remains as an open, challenging but a meaningful problem. This work attempts to address this problem by defining the concept of task-level performance and developing an approach for systematic assessment based on a proposed task representation for performance quantification of different task plans and robotic configurations. Starting from productivity which is one of the main concern for manufacturers, this work introduces a methodology for quantitative assessment of any given robotic configuration and task handling. Such a method is useful for comparing the productivity of different robotic configurations and evaluating the worthiness of updated solution of a sub-task through observing improvements in productivity. Implemented and tested on a peg-in-hole task with the different level of difficulty through single-arm and dual-arm manipulation on a dual-arm robot, the methodology has demonstrated encouraging results. Finally, the connection of performance assessment from task-level to economy-level is presented.


Industrial robot Task planning Performance evaluation 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.



  1. 1.
    Poole, H.H.: Fundamentals of Robotics Engineering. Van Nostrand Reinhold, New York (1989)CrossRefGoogle Scholar
  2. 2.
    Park K.T., Park C.H., Shin Y.J.: Performance evaluation of industrial dual-arm robot. In: 2008 International Conference on Smart Manufacturing Application, pp. 437—440. IEEE, Gyeonggi-do (2008)Google Scholar
  3. 3.
    Whitney, D.E.: Mechanical Assemblies: Their Design, Manufacture and Role in Product Development. Oxford University Press, New York (2004)Google Scholar
  4. 4.
    Salisbury, D.E., Craig, J.J.: Articulated hands: Force control and kinematic issues. Int. J. Robot. Res. 1 (1), 4—17 (1982)CrossRefGoogle Scholar
  5. 5.
    Yoshikawa, T., Craig, J.J.: Dynamic manipulability of robot manipulators. Trans. Soc. Instrum. Control Eng. 21(9), 970–975 (1985)CrossRefGoogle Scholar
  6. 6.
    Bowling, A., Khatib, O.: The dynamic capability equations: a new tool for analyzing robotic manipulator performance. Trans. Robot. 21(1), 115—123 (2005)Google Scholar
  7. 7.
    Gosselin, C., Angeles, J.: A global performance index for the kinematic optimization of robotic manipulators. J. Mech. Des. 113(3), 220–226 (1991)CrossRefGoogle Scholar
  8. 8.
    Graettinger, T.J., Krogh, B.H.: The acceleration radius: a global performance measure for robotic manipulators. J. Robot. Autom. 4(1), 60–69 (1988)CrossRefGoogle Scholar
  9. 9.
    Wan, W., Harada, K.: Developing and comparing single-arm and dual-arm regrasp. Robot. Autom. Lett. 1(1), 243–250 (2016)CrossRefGoogle Scholar
  10. 10.
    Sedighi, K.H., Ashenayi, K., Manikas, T.W., Wainwright, R.L., Tai, H.M.: Autonomous local path planning for a mobile robot using a genetic algorithm. In: 2004 Congress on Evolutionary Computation, pp. 1338–1345. IEEE, Portland (2004)Google Scholar
  11. 11.
    Crandall, J.W., Nielsen, C.W., Goodrich, M.A.: Towards predicting robot team performance. In: 2003 International Conference on Systems, Man and Cybernetics, pp. 906–911. IEEE, Washington (2003)Google Scholar
  12. 12.
    De Visser, E., Parasuraman, R., Freedy, A., Freedy, E., Weltman, G.: A comprehensive methodology for assessing human-robot team performance for use in training and simulation. In: 2006 the Human Factors and Ergonomics Society Annual Meeting, pp. 2639–2643. SAGE, Los Angeles (2006)Google Scholar
  13. 13.
    Balakirsky, S., Scrapper, C., Carpin, S., Lewis, M.: Usarsim: Providing a Framework for Multi-Robot Performance Evaluation. In: 2006 Performance Metrics for Intelligent Systems, pp. 2639–2643. Gaithersburg (2006)Google Scholar
  14. 14.
    Lampe, A., Chatila, R.: Performance Measure for the Evaluation of Mobile Robot Autonomy. In: 2006 International Conference on Robotics and Automation, pp. 4057–4062, Orlando (2006)Google Scholar
  15. 15.
    Schreckenghost, D., Fong, T., Utz, H., Milam, T.: Measuring robot performance in real-time for nasa robotic reconnaissance operations. In: The 9th Workshop on Performance Metrics for Intelligent Systems, pp. 194—202. ACM, Gaithersburg (2009)Google Scholar
  16. 16.
    National Institute of Standards and Technology: Performance metrics and benchmarks to advance the state of robotic assembly. Available at: (accessed 9 May 2018) (2016)
  17. 17.
    Latombe, J.C.: Robot motion planning. Springer Science & Business Media, Berlin (2012)Google Scholar
  18. 18.
    Koller, D., Friedman, N.: Probabilistic Graphical Models : Principles and Techniques. The MIT Press, Cambridge (2009)zbMATHGoogle Scholar
  19. 19.
    Viswanadham, N., Narahari, Y.: Performance Modeling of Automated Manufacturing Systems. NJ, Prentice Hall (2009)zbMATHGoogle Scholar
  20. 20.
    Durrett, R., Narahari, Y.: Probability: Theory and Examples. Cambridge University Press, New York (2009)Google Scholar
  21. 21.
    Feller, W.: On the Theory of Stochastic Processes, with Particular Reference to Applications. In: Schilling, R., Vondraček, Z., Woyczyński, W. (eds.) Selected Papers I, pp 769–798. Springer (2015)Google Scholar
  22. 22.
    Ching, W., Ng, M.: Markov Chains: Models Algorithms and Applications. Springer Science & Business Media, Boston (2006)zbMATHGoogle Scholar
  23. 23.
    Montgomery, D.C.: Design and analysis of experiments. Wiley, New York (2017)Google Scholar
  24. 24.
    Slamani, M., Nubiola, A., Bonev, I.: Assessment of the positioning performance of an industrial robot. Ind. Robot: Int. J. 39(1), 57–68 (2012)CrossRefGoogle Scholar
  25. 25.
    Steinfeld, A., Fong, T., Kaber, D., Lewis, M., Scholtz, J., Schultz, A., Goodrich, M.: Common Metrics for Human-Robot Interaction. In: The 1St ACM SIGCHI/SIGART Conference on Human-Robot Interaction, pp. 33–40, Salt Lake City (2006)Google Scholar
  26. 26.
    Olsen, D.R., Goodrich, M.A.: Metrics for evaluating human-robot interaction. Proceedings of PERMIS, Gaithersburg, pp. 4 (2003)Google Scholar
  27. 27.
    Weng, C.Y., Tan, W.C., Chen, I.M.: A Survey of Dual-Arm Robotic Issues on Assembly Tasks. In: ROMANSY 22-Robot Design, Dynamics and Control, pp. 474–480. Springer, Cham (2018)Google Scholar
  28. 28.
    Weng, C.Y., Chen, I.M.: The Task-Level Evaluation Model for a Flexible Assembly Task with an Industrial Dual-Arm Robot. In: 2017 IEEE International Conference on Cybernetics and Intelligent Systems (CIS) and IEEE Conference on Robotics, Automation and Mechatronics (RAM), pp. 356-360, Ningbo, China (2017)Google Scholar

Copyright information

© Springer Nature B.V. 2019

Authors and Affiliations

  1. 1.Robotics Research CentreNanyang Technological UniversitySingaporeSingapore

Personalised recommendations