Autonomous Robots

, Volume 42, Issue 7, pp 1459–1475 | Cite as

Four aspects of building robotic systems: lessons from the Amazon Picking Challenge 2015

  • Clemens Eppner
  • Sebastian Höfer
  • Rico Jonschkowski
  • Roberto Martín-Martín
  • Arne SieverlingEmail author
  • Vincent Wall
  • Oliver Brock
Part of the following topical collections:
  1. Special Issue on Robotics: Science and Systems 2016


We describe the winning entry to the Amazon Picking Challenge  2015. From the experience of building this system and competing in the Amazon Picking Challenge, we derive several conclusions: (1) We suggest to characterize robotic system building along four key aspects, each of them spanning a spectrum of solutions—modularity versus integration, generality versus assumptions, computation versus embodiment, and planning versus feedback. (2) To understand which region of each spectrum most adequately addresses which robotic problem, we must explore the full spectrum of possible approaches. To achieve this, our community should agree on key aspects that characterize the solution space of robotic systems. (3) For manipulation problems in unstructured environments, certain regions of each spectrum match the problem most adequately, and should be exploited further. This is supported by the fact that our solution deviated from the majority of the other 2015 challenge entries along each of the spectra.


Robotic systems Amazon Picking Challenge Warehouse automation Mobile manipulation 



We would like to thank Barrett Technology for their support and our team members Raphael Deimel, Roman Kolbert, Gabriel Le Roux, and Wolf Schaarschmidt, who helped creating the winning system.


  1. Aloimonos, J., Weiss, I., & Bandyopadhyay, A. (1988). Active vision. International Journal of Computer Vision, 1(4), 333–356.CrossRefGoogle Scholar
  2. Atkeson, C. G., Babu, B. P. W., Banerjee, N., Berenson, D., Bove, C. P., Cui, X., DeDonato, M., Du, R., Feng, S., & Franklin, P., et al. (2015). No falls, no resets: Reliable humanoid behavior in the DARPA robotics challenge. In IEEE-RAS 15th international conference on humanoid robots (Humanoids) (pp 623–630). IEEE.Google Scholar
  3. Baldwin, C. Y., & Clark, K. B. (2000). Design rules: The power of modularity (Vol. 1). Cambridge: MIT Press.Google Scholar
  4. Bertsekas, D. P. (1995). Dynamic programming and optimal control. Belmont: Athena Scientific.zbMATHGoogle Scholar
  5. Bohg, J., Hausman, K., Sankaran, B., Brock, O., Kragic, D., Schaal, S., et al. (2017). Interactive perception: Leveraging action in perception and perception in action. IEEE Transactions on Robotics, 33(6), 1273–1291.CrossRefGoogle Scholar
  6. Bradski, G. (2000). The openCV library. Dr Dobb’s Journal: Software Tools for the Professional Programmer, 25(11), 120–123.Google Scholar
  7. Brooks, F. P. (1995). The Mythical Man-Month: Essays on software engineering. Boston: Addison-Wesley Longman.Google Scholar
  8. Brooks, R. A. (1990). Elephants don’t play chess. Robotics and Autonomous Systems, 6(1), 3–15.CrossRefGoogle Scholar
  9. Brooks, R. A., Breazeal, C., Irie, R., Kemp, C. C., Marjanovic, M., Scassellati, B., & Williamson, M. M. (1998). Alternative essences of intelligence. In: AAAI/IAAI (pp. 961–968).Google Scholar
  10. Brown, E., Rodenberg, N., Amend, J., Mozeika, A., Steltz, E., Zakin, M. R., et al. (2010). Universal robotic gripper based on the jamming of granular material. Proceedings of the National Academy of Sciences, 107(44), 18,809–18,814.CrossRefGoogle Scholar
  11. Buehler, M., Iagnemma, K., & Singh, S. (2007). The 2005 DARPA grand challenge: The great robot race. Berlin: Springer.CrossRefGoogle Scholar
  12. Cohen, P. R. (1996). Empirical methods for artificial intelligence. IEEE Intelligent Systems, 11(6), 88.Google Scholar
  13. Coleman, D. (2015). MoveIt! strengths, weaknesses, and developer insight, presentation at ROSCon. Accessed 16 Nov 2017.
  14. Correll, N., Bekris, K. E., Berenson, D., Brock, O., Causo, A., Hauser, K., et al. (2018). Analysis and observations from the first amazon picking challenge. IEEE Transactions on Automation Science and Engineering, 15(1), 172–188.CrossRefGoogle Scholar
  15. Deimel, R., & Brock, O. (2016). A novel type of compliant and underactuated robotic hand for dexterous grasping. The International Journal of Robotics Research, 35(1–3), 161–185.CrossRefGoogle Scholar
  16. Dollar, A. M., & Howe, R. D. (2010). The highly adaptive SDM hand: Design and performance evaluation. The International Journal of Robotics Research, 29(5), 585–597.CrossRefGoogle Scholar
  17. Egerstedt, M. (2000). Behavior based robotics using hybrid automata. In Hybrid Systems: computation and control (pp 103–116). Springer.Google Scholar
  18. Eppner, C., Deimel, R., Álvarez Ruiz, J., Maertens, M., & Brock, O. (2015). Exploitation of environmental constraints in human and robotic grasping. The International Journal of Robotics Research, 34(7), 1021–1038.CrossRefGoogle Scholar
  19. Eppner, C., Höfer, S., Jonschkowski, R., Martín-Martín, R., Sieverling, A., Wall, V., & Brock, O. (2016). Lessons from the Amazon Picking Challenge: Four aspects of building robotic systems. In Proceedings of Robotics: Science and Systems.Google Scholar
  20. Espiau, B., Chaumette, F., & Rives, P. (1992). A new approach to visual servoing in robotics. IEEE Transactions on Robotics and Automation, 8(3), 313–326.CrossRefGoogle Scholar
  21. Hastie, T., Tibshirani, R., Friedman, J., & Franklin, J. (2005). The elements of statistical learning: Data mining, inference and prediction. The Mathematical Intelligencer, 27(2), 83–85.Google Scholar
  22. Hawes, N., Zillich, M., & Jensfelt, P. (2010). Lessons learnt from scenario-based integration. In Cognitive Systems (pp 423–438). Springer.Google Scholar
  23. Hernandez, C., Bharatheesha, M., Ko, W., Gaiser, H., Tan, J., van Deurzen, K., de Vries, M., Mil, B. V., van Egmond, J., Burger, R., Morariu, M., Ju, J., Gerrmann, X., Ensing, R., van Frankenhuyzen, J., & Wisse, M. (2016). Team delft’s robot winner of the Amazon Picking Challenge 2016. CoRR arXiv:1610.05514.
  24. Hollerbach, J., Khalil, W., & Gautier, M. (2008). Model identification. In B. Siciliano & O. Khatib (Eds.), Springer handbook of robotics (pp. 321–344). Berlin: Springer.CrossRefGoogle Scholar
  25. Holmberg, R., & Khatib, O. (2000). Development and control of a holonomic mobile robot for mobile manipulation tasks. The International Journal of Robotics Research, 19(11), 1066–1074.CrossRefzbMATHGoogle Scholar
  26. Ijspeert, A. J., Nakanishi, J., Hoffmann, H., Pastor, P., & Schaal, S. (2013). Dynamical movement primitives: Learning attractor models for motor behaviors. Neural Computation, 25(2), 328–373.MathSciNetCrossRefzbMATHGoogle Scholar
  27. Johnson, M., Shrewsbury, B., Bertrand, S., Wu, T., Duran, D., Floyd, M., et al. (2015). Team IHMC’s lessons learned from the DARPA robotics challenge trials. Journal of Field Robotics, 32(2), 192–208.CrossRefGoogle Scholar
  28. Jonschkowski, R., & Brock, O. (2015). Learning state representations with robotic priors. Autonomous Robots, 39(3), 407–428.CrossRefGoogle Scholar
  29. Jonschkowski, R., Eppner, C., Höfer, S., Martín-Martín, R., & Brock, O. (2016). Probabilistic multi-class segmentation for the Amazon Picking Challenge. In IEEE/RSJ international conference on intelligent robots and systems (pp. 822–830).Google Scholar
  30. Kamiya, Y. (2016). Team preferred networks. In Talk at workshop on automation for warehouse logistics (CASE2016). Accessed 16 Feb 2016.
  31. Katz, D., & Brock, O. (2011). A factorization approach to manipulation in unstructured environments. In Robotics Research (pp. 285–300). Springer.Google Scholar
  32. Laumond, J. P. (2006). Kineo CAM: A success story of motion planning algorithms. IEEE Robotics & Automation Magazine, 13(2), 90–93.CrossRefGoogle Scholar
  33. LaValle, S. M. (2006). Planning algorithms. Cambridge: Cambridge University Press.CrossRefzbMATHGoogle Scholar
  34. Lozano-Perez, T., Mason, M. T., & Taylor, R. H. (1984). Automatic synthesis of fine-motion strategies for robots. The International Journal of Robotics Research, 3(1), 3–24.CrossRefGoogle Scholar
  35. Marder-Eppstein, E., Berger, E., Foote, T., Gerkey, B., & Konolige, K. (2010). The office marathon: Robust navigation in an indoor office environment. In IEEE International Conference on Robotics and Automation (ICRA) (pp. 300–307).Google Scholar
  36. Marr, D. (1982). Vision: A computational approach. San Francisco: Freeman & Co.Google Scholar
  37. Martín-Martín, R., & Brock, O. (2014). Online interactive perception of articulated objects with multi-level recursive estimation based on task-specific priors. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (pp. 2494–2501).Google Scholar
  38. Martín-Martín, R., Höfer, S., & Brock, O. (2016). An integrated approach to visual perception of articulated objects. In IEEE International Conference on Robotics and Automation (ICRA) (pp. 5091–5097).Google Scholar
  39. Mason, M. T., Rodriguez, A., Srinivasa, S. S., & Vazquez, A. S. (2012). Autonomous manipulation with a general-purpose simple hand. The International Journal of Robotics Research, 31(5), 688–703.CrossRefGoogle Scholar
  40. Milan, A. (2016). Team NimbRo. In: Talk at Workshop on Automation for Warehouse Logistics (CASE2016). Accessed 16 Feb 2016.
  41. Miller, A. T., & Allen, P. K. (2004). GraspIt! a versatile simulator for robotic grasping. IEEE Robotics & Automation Magazine, 11(4), 110–122.CrossRefGoogle Scholar
  42. Morrison, D., Tow, A. W., McTaggart, M., Smith, R., Kelly-Boxall, N., Wade-McCue, S., Erskine, J., Grinover, R., Gurman, A., Hunn, T., Lee, D., Milan, A., Pham, T., Rallos, G., Razjigaev, A., Rowntree, T., Vijay, K., Zhuang, Z., Lehnert, C. F., Reid, I. D., Corke, P., & Leitner, J. (2017). Cartman: The low-cost cartesian manipulator that won the amazon robotics challenge. CoRR arXiv:1709.06283.
  43. Nakanishi, J., Cory, R., Mistry, M., Peters, J., & Schaal, S. (2008). Operational space control: A theoretical and empirical comparison. The International Journal of Robotics Research, 27(6), 737–757.CrossRefGoogle Scholar
  44. Papadimitriou, C. H., & Tsitsiklis, J. N. (1987). The complexity of Markov decision processes. Mathematics of Operations Research, 12(3), 441–450.MathSciNetCrossRefzbMATHGoogle Scholar
  45. Pfeifer, R., & Gómez, G. (2009). Morphological computation: Connecting body, brain, and environment. In Creating brain-like intelligence (pp. 66–83). Springer.Google Scholar
  46. Quigley, M., Conley, K., Gerkey, B., Faust, J., Foote, T., Leibs, J., Wheeler, R., & Ng, A. Y. (2009). ROS: An open-source robot operating system. In ICRA workshop on open source software.Google Scholar
  47. Rennie, C., Shome, R., Bekris, K. E., & De Souza, A. F. (2016). A dataset for improved rgbd-based object detection and pose estimation for warehouse pick-and-place. IEEE Robotics and Automation Letters, 1(2), 1179–1185.CrossRefGoogle Scholar
  48. Rickert, M., Sieverling, A., & Brock, O. (2014). Balancing exploration and exploitation in sampling-based motion planning. IEEE Transactions on Robotics, 30(6), 1305–1317.CrossRefGoogle Scholar
  49. Rooks, B. (2006). The harmonious robot. Industrial Robot: An International Journal, 33(2), 125–130.CrossRefGoogle Scholar
  50. Rusu, R. B., & Cousins, S. (2011). 3D is here: Point Cloud Library (PCL). In IEEE International Conference on Robotics and Automation (ICRA) (pp. 1–4).Google Scholar
  51. Scholz, J., Levihn, M., Isbell, C., & Wingate, D. (2014). A physics-based model prior for object-oriented MDPs. In Proceedings of the 31st International Conference on Machine Learning (ICML-14) (pp. 1089–1097).Google Scholar
  52. Schwaber, K. (2004). Agile project management with Scrum. Redmond: Microsoft Press.zbMATHGoogle Scholar
  53. Schwarz, M., Milan, A., Lenz, C., Munoz, A., Periyasamy, A. S., Schreiber, M., Schüller, S., & Behnke, S. (2017). NimbRo picking: Versatile part handling for warehouse automation. In IEEE International Conference on Robotics and Automation (ICRA).Google Scholar
  54. Schwarz, M., Milan, A., Periyasamy, A. S., & Behnke, S. (2018). RGB-D object detection and semantic segmentation for autonomous manipulation in clutter. The International Journal of Robotics Research, 37(4–5), 437–451.
  55. Stulp, F., & Sigaud, O. (2013). Robot skill learning: From reinforcement learning to evolution strategies. Paladyn, Journal of Behavioral Robotics, 4(1), 49–61.CrossRefGoogle Scholar
  56. Sucan, I. I., & Chitta, S. (2016). MoveIt. Accessed 16 Nov 2017.
  57. Thrun, S., Burgard, W., & Fox, D. (2005). Probabilistic robotics. Cambridge: MIT Press.zbMATHGoogle Scholar
  58. Ulrich, K. (1995). The role of product architecture in the manufacturing firm. Research Policy, 24(3), 419–440.CrossRefGoogle Scholar
  59. van Gigch, J. P. (1991). System design modeling and metamodeling. Berlin: Springer.CrossRefGoogle Scholar
  60. Wolpert, D. H. (1996). The lack of a priori distinctions between learning algorithms. Neural Computation, 8(7), 1341–1390.CrossRefGoogle Scholar
  61. Wolpert, D. H., & Macready, W. G. (1997). No free lunch theorems for optimization. IEEE Transactions on Evolutionary Computation, 1(1), 67–82.CrossRefGoogle Scholar
  62. XR4000. (2015). Nomadic technologies inc. XR4000. Accessed 29 Jan 2016.
  63. Yu, K. T., Fazeli, N., Chavan-Dafle, N., Taylor, O., Donlon, E., Lankenau, G. D., & Rodriguez, A. (2016). A summary of team MIT’s approach to the Amazon Picking Challenge 2015. ArXiv e-prints arXiv:1604.03639.
  64. Yu, P. (2016). Team MIT. In Talk at Workshop on Automation for Warehouse Logistics (CASE2016). Accessed 16 Feb 2016.
  65. Zeng, A., Yu, K. T., Song, S., Suo, D., Walker, E., Rodriguez, A., & Xiao, J. (2017). Multi-view self-supervised deep learning for 6d pose estimation in the Amazon Picking Challenge. In IEEE International Conference on Robotics and Automation (ICRA) (pp. 1386–1383).Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Robotics and Biology LaboratoryBerlinGermany

Personalised recommendations