A Taxonomy of Benchmark Tasks for Robot Manipulation

  • Ana Huamán QuispeEmail author
  • Heni Ben Amor
  • Henrik I. Christensen
Part of the Springer Proceedings in Advanced Robotics book series (SPAR, volume 2)


This paper presents a taxonomy of benchmark manipulation tasks for service robots. Our contributions are threefold: (1) A review of relevant literature regarding manipulation tests in the robotics domain and related fields, such as physical therapy, assistive technologies and prosthetics. (2) Guidelines to design useful testing protocols to evaluate manipulation performance. (3) A proposed general taxonomy of benchmark manipulation tasks and sample tests per each class.


Benchmark Tasks Robot Manipulation Manipulation Tests Amazon Picking Challenge (APC) Grasp Quality Measures 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Aaron, D., Jansen, C.: Development of the functional dexterity test (FDT): construction, validity, reliability, and normative data. J. Hand Ther. 16(1), 12–21 (2003)CrossRefGoogle Scholar
  2. 2.
    Balasubramanian, R., Xu, L., Brook, P., Smith, J., Matsuoka, Y.: Physical human interactive guidance: identifying grasping principles from human-planned grasps. The Human Hand as an Inspiration for Robot Hand Development. Springer, Switzerland (2014)CrossRefGoogle Scholar
  3. 3.
    Barreca, S., Gowland, C., Stratford, P., Huijbregts, M., Griffiths, J., Torresin, W., Dunkley, M., Miller, P., Masters, L.: Development of the chedoke arm and hand activity inventory: theoretical constructs, item generation, and selection. Topics Stroke Rehabil. 11(4), 31–42 (2004)CrossRefGoogle Scholar
  4. 4.
    Bohg, J., Morales, A., Asfour, T., Kragic, D.: Data-driven grasp synthesis—a survey. IEEE Trans. Robot. 30, (2013)Google Scholar
  5. 5.
    Bryden, P., Roy, E.: A new method of administering the grooved pegboard test: performance as a function of handedness and sex. Brain Cogn. 58(3), xxx (2005)CrossRefGoogle Scholar
  6. 6.
    Choi, Y., Deyle, T., Chen, T., Glass, J., Kemp, C.: A list of household objects for robotic retrieval prioritized by people with ALS. In: IEEE International Conference on Rehabilitation Robotics (2009)Google Scholar
  7. 7.
    Collins, K., Palmer, A., Rathmill, K.: The development of a European benchmark for the comparison of assembly robot programming systems. Robot Technology and Applications. Springer, Heidelberg (1985)Google Scholar
  8. 8.
    Dantam, N., Ben Amor, H., Christensen, H., Stilman, M.: Online multi-camera registration for bimanual workspace trajectories. In: HUMANOIDS (2014)Google Scholar
  9. 9.
    Deimel, R., Brock, O.: A novel type of compliant, underactuated robotic hand for dexterous grasping. In: Proceedings of Robotics: Science and Systems, pp. 1687–1692 (2014)Google Scholar
  10. 10.
    Desrosiers, J., Hébert, R., Dutil, E., Bravo, G.: Development and reliability of an upper extremity function test for the elderly: the TEMPA. Can. J. Occup. Ther. 60(1), 9–16 (1993)CrossRefGoogle Scholar
  11. 11.
    Feix, T., Bullock, I., Dollar, A.: Analysis of human grasping behavior: correlating tasks, objects and grasps. IEEE Trans. Haptics 7, 430–441 (2014)CrossRefGoogle Scholar
  12. 12.
    Feix, T., Bullock, I., Dollar, A.: Analysis of human grasping behavior: object characteristics and grasp type. IEEE Trans. Haptics 7, 311–323 (2014)CrossRefGoogle Scholar
  13. 13.
    Feix, T., Romero, J., Ek, C., Schmiedmayer, H., Kragic, D.: A metric for comparing the anthropomorphic motion capability of artificial hands. IEEE Trans. Robot. 29(1), 82–93 (2013)CrossRefGoogle Scholar
  14. 14.
    Ferrari, C., Canny, J.: Planning optimal grasps. In: ICRA (1992)Google Scholar
  15. 15.
    Gloss, D., Wardle, M.: Use of the minnesota rate of manipulation test for disability evaluation. Percept. Mot. Skills 55(2), 527–532 (1982)CrossRefGoogle Scholar
  16. 16.
    Grebenstein, M.: The awiwi hand: an artificial hand for the DLR hand arm system. Approaching Human Performance, pp. 65–130. Springer, Switzerland (2014)CrossRefGoogle Scholar
  17. 17.
    Grunwald, G., Borst, C., Zöllner, J.E.A.: Benchmarking dexterous dual-arm/hand robotic manipulation. In: IROS Workshop on Performance Evaluation and Benchmarking (2008)Google Scholar
  18. 18.
    Hackett, D., Pippine, J., Watson, A., Sullivan, C., Pratt, G.: An overview of the DARPA autonomous robotic manipulation (ARM) program. J. Robot. Soc. Jpn. 31(4), 326–329 (2013)CrossRefGoogle Scholar
  19. 19.
    Iossifidis, I., Lawitzky, G., Knoop, S., Zöllner, R.: Towards benchmarking of domestic robotic assistants. Advances in Human-Robot Interaction. Springer, Heidelberg (2005)Google Scholar
  20. 20.
    Jebsen, R., Taylor, N., Trieschmann, R., Trotter, M., Howard, L.: An objective and standardized test of hand function. Arch. Phys. Med. Rehabil. 50(6), 311 (1969)Google Scholar
  21. 21.
    Kapandji, A.: Clinical test of apposition and counter-apposition of the thumb. Annales de chirurgie de la main: organe officiel des societes de chirurgie de la main 5(1) (1985)Google Scholar
  22. 22.
    Kim, J., Iwamoto, K., Kuffner, J., Ota, Y., Pollard, N.: Physically based grasp quality evaluation under pose uncertainty. IEEE Trans. Robot. 29, 1424 (2013)CrossRefGoogle Scholar
  23. 23.
    Kopp, B., Kunkel, A., Flor, H., Platz, T., Rose, U., Mauritz, K., Gresser, K., McCulloch, K., Taub, E.: The arm motor ability test: reliability, validity, and sensitivity to change of an instrument for assessing disabilities in activities of daily living. Arch. Phys. Med. Rehabil. 78(6), 615–620 (1997)CrossRefGoogle Scholar
  24. 24.
    Kyberd, P., Murgia, A., Gasson, M., Tjerks, T., Metcalf, C., Chappell, P., Warwick, K., Lawson, S., Barnhill, T.: Case studies to demonstrate the range of applications of the Southampton Hand Assessment Procedure. Br. J. Occup. Ther. 72(5), 212–218 (2009)CrossRefGoogle Scholar
  25. 25.
    Leidner, D., Borst, C., Hirzinger, G.: Things are made for what they are: solving manipulation tasks by using functional object classes. In: HUMANOIDS (2012)Google Scholar
  26. 26.
    Light, C.M., Chappell, P.H., Kyberd, P.: Establishing a standardized clinical assessment tool of pathologic and prosthetic hand function: normative data, reliability, and validity. Arch. Phys. Med. Rehabil. 83(6), 776–783 (2002)CrossRefGoogle Scholar
  27. 27.
    Lin, S., Chang, J., Chen, P., Mao, H.: Hand function measures for burn patients: a literature review. Burns (J. Int. Soc. Burn Inj.) 39(1), 16–23 (2013)CrossRefGoogle Scholar
  28. 28.
    Matheus, K., Dollar, A.: Benchmarking grasping and manipulation: properties of the objects of daily living. In: IROS (2010)Google Scholar
  29. 29.
    Mathiowetz, V., Weber, K., Volland, G., Kashman, N.: Reliability and validity of grip and pinch strength evaluations. J. Hand Surg. 9(2), 222–226 (1984)CrossRefGoogle Scholar
  30. 30.
    Mathiowetz, V., Volland, G., Kashman, N., Weber, K.: Adult norms for the box and block test of manual dexterity. Am. J. Occup. Ther. 39(6), 386–391 (1985)CrossRefGoogle Scholar
  31. 31.
    Mathiowetz, V., Rogers, S., Dowe-Keval, M., Donahoe, L., Rennells, C.: The purdue pegboard: norms for 14-to 19-year-olds. Am. J. Occup. Ther. 40(3), 174–179 (1986)CrossRefGoogle Scholar
  32. 32.
    Meeussen, W., Wise, M., Glaser, S., Chitta, S., McGann, C., Mihelich, P., Marder-Eppstein, E., Muja, M., Eruhimov, V., Foote, T., et al.: Autonomous door opening and plugging in with a personal robot. In: ICRA, pp. 729–736 (2010)Google Scholar
  33. 33.
    Morales, A., Chinellato, E., Sanz, P., Del Pobil, A., Fagg, A.H.: Learning to predict grasp reliability for a multifinger robot hand by using visual features. In: AISC (2004)Google Scholar
  34. 34.
    Mukai, T., Hirano, S., Nakashima, H., Kato, Y., Sakaida, Y., Guo, S., Hosoe, S.: Development of a nursing-care assistant robot RIBA that can lift a human in its arms. In: IROS (2010)Google Scholar
  35. 35.
    Ng, C., Ho, D., Chow, S.: The Moberg pickup test: results of testing with a standard protocol. J. Hand Ther. 12(4), 309–312 (1999)CrossRefGoogle Scholar
  36. 36.
    Poole, J., Burtner, P., Torres, T., McMullen, C., Markham, A., Marcum, M., Anderson, J., Qualls, C.: Measuring dexterity in children using the nine-hole peg test. J. Hand Ther. 18(3), 348–351 (2005)CrossRefGoogle Scholar
  37. 37.
    Roa, M., Hertkorn, K., Zacharias, F., Borst, C., Hirzinger, G.: Graspability map: a tool for evaluating grasp capabilities. In: IEEE/RSJ IROS, pp. 1768–1774 (2011)Google Scholar
  38. 38.
    Schoneveld, K., Wittink, H., Takken, T.: Clinimetric evaluation of measurement tools used in hand therapy to assess activity and participation. J. Hand Ther. 22(3), 221–236 (2009)CrossRefGoogle Scholar
  39. 39.
    Siciliano, B.: Advanced Bimanual Manipulation: Results from The DEXMART Project, vol. 80. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  40. 40.
    Sollerman, C., Ejeskär, A.: Sollerman hand function test: a standardised method and its use in tetraplegic patients. Scand. J. Plast. Reconstr. Surg. Hand Surg. 29(2), 167–176 (1995)CrossRefGoogle Scholar
  41. 41.
    The Amazon Picking Challenge. (2014)
  42. 42.
    Tsui, K., Feil-Seifer, D., Matarić, M.J., Yanco, H.: Performance evaluation methods for assistive robotic technology. Performance Evaluation and Benchmarking of Intelligent Systems, pp. 41–66. Springer, US (2009)CrossRefGoogle Scholar
  43. 43.
    van Lankveld, W., van’t Pad Bosch, P., Bakker, J., Terwindt, S., Franssen, M.: Sequential occupational dexterity assessment (SODA): a new test to measure hand disability. J. Hand Ther. 9(1), 27–32 (1996)Google Scholar
  44. 44.
    Williams, M., Hadler, N., Earp, J.: Manual ability as a marker of dependency in geriatric women. J. Chronic Dis. 35(2), 115–122 (1982)CrossRefGoogle Scholar
  45. 45.
    Wisspeintner, T., Van Der Zant, T., Iocchi, L., Schiffer, S.: Robocup@Home: scientific competition and benchmarking for domestic service robots. Interact. Stud. 10(3), 392–426 (2009)CrossRefGoogle Scholar
  46. 46.
    Wolf, S., Thompson, P., Morris, D., D.K., Winstein, C., Taub, E., Giuliani, C., Pearson, S.: The EXCITE trial: attributes of the wolf motor function test in patients with subacute stroke. Neurorehabil. Neural Repair 19(3), 194–205 (2005)Google Scholar
  47. 47.
    Workshop on Autonomous Grasping and Manipulation: An Open Challenge. (2014)
  48. 48.
    Yozbatiran, N., Der-Yeghiaian, L., Cramer, S.: A standardized approach to performing the action research arm test. Neurorehabil. Neural Repair 22(1), 78–90 (2008)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2018

Authors and Affiliations

  • Ana Huamán Quispe
    • 1
    Email author
  • Heni Ben Amor
    • 1
  • Henrik I. Christensen
    • 1
  1. 1.Institute for Robotics and Intelligent MachinesGeorgia Institute of TechnologyAtlantaUSA

Personalised recommendations