Four aspects of building robotic systems: lessons from the Amazon Picking Challenge 2015


We describe the winning entry to the Amazon Picking Challenge  2015. From the experience of building this system and competing in the Amazon Picking Challenge, we derive several conclusions: (1) We suggest to characterize robotic system building along four key aspects, each of them spanning a spectrum of solutions—modularity versus integration, generality versus assumptions, computation versus embodiment, and planning versus feedback. (2) To understand which region of each spectrum most adequately addresses which robotic problem, we must explore the full spectrum of possible approaches. To achieve this, our community should agree on key aspects that characterize the solution space of robotic systems. (3) For manipulation problems in unstructured environments, certain regions of each spectrum match the problem most adequately, and should be exploited further. This is supported by the fact that our solution deviated from the majority of the other 2015 challenge entries along each of the spectra.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12


  1. Aloimonos, J., Weiss, I., & Bandyopadhyay, A. (1988). Active vision. International Journal of Computer Vision, 1(4), 333–356.

    Article  Google Scholar 

  2. Atkeson, C. G., Babu, B. P. W., Banerjee, N., Berenson, D., Bove, C. P., Cui, X., DeDonato, M., Du, R., Feng, S., & Franklin, P., et al. (2015). No falls, no resets: Reliable humanoid behavior in the DARPA robotics challenge. In IEEE-RAS 15th international conference on humanoid robots (Humanoids) (pp 623–630). IEEE.

  3. Baldwin, C. Y., & Clark, K. B. (2000). Design rules: The power of modularity (Vol. 1). Cambridge: MIT Press.

    Google Scholar 

  4. Bertsekas, D. P. (1995). Dynamic programming and optimal control. Belmont: Athena Scientific.

    Google Scholar 

  5. Bohg, J., Hausman, K., Sankaran, B., Brock, O., Kragic, D., Schaal, S., et al. (2017). Interactive perception: Leveraging action in perception and perception in action. IEEE Transactions on Robotics, 33(6), 1273–1291.

    Article  Google Scholar 

  6. Bradski, G. (2000). The openCV library. Dr Dobb’s Journal: Software Tools for the Professional Programmer, 25(11), 120–123.

    Google Scholar 

  7. Brooks, F. P. (1995). The Mythical Man-Month: Essays on software engineering. Boston: Addison-Wesley Longman.

    Google Scholar 

  8. Brooks, R. A. (1990). Elephants don’t play chess. Robotics and Autonomous Systems, 6(1), 3–15.

    Article  Google Scholar 

  9. Brooks, R. A., Breazeal, C., Irie, R., Kemp, C. C., Marjanovic, M., Scassellati, B., & Williamson, M. M. (1998). Alternative essences of intelligence. In: AAAI/IAAI (pp. 961–968).

  10. Brown, E., Rodenberg, N., Amend, J., Mozeika, A., Steltz, E., Zakin, M. R., et al. (2010). Universal robotic gripper based on the jamming of granular material. Proceedings of the National Academy of Sciences, 107(44), 18,809–18,814.

    Article  Google Scholar 

  11. Buehler, M., Iagnemma, K., & Singh, S. (2007). The 2005 DARPA grand challenge: The great robot race. Berlin: Springer.

    Google Scholar 

  12. Cohen, P. R. (1996). Empirical methods for artificial intelligence. IEEE Intelligent Systems, 11(6), 88.

    Google Scholar 

  13. Coleman, D. (2015). MoveIt! strengths, weaknesses, and developer insight, presentation at ROSCon. Accessed 16 Nov 2017.

  14. Correll, N., Bekris, K. E., Berenson, D., Brock, O., Causo, A., Hauser, K., et al. (2018). Analysis and observations from the first amazon picking challenge. IEEE Transactions on Automation Science and Engineering, 15(1), 172–188.

    Article  Google Scholar 

  15. Deimel, R., & Brock, O. (2016). A novel type of compliant and underactuated robotic hand for dexterous grasping. The International Journal of Robotics Research, 35(1–3), 161–185.

    Article  Google Scholar 

  16. Dollar, A. M., & Howe, R. D. (2010). The highly adaptive SDM hand: Design and performance evaluation. The International Journal of Robotics Research, 29(5), 585–597.

    Article  Google Scholar 

  17. Egerstedt, M. (2000). Behavior based robotics using hybrid automata. In Hybrid Systems: computation and control (pp 103–116). Springer.

  18. Eppner, C., Deimel, R., Álvarez Ruiz, J., Maertens, M., & Brock, O. (2015). Exploitation of environmental constraints in human and robotic grasping. The International Journal of Robotics Research, 34(7), 1021–1038.

    Article  Google Scholar 

  19. Eppner, C., Höfer, S., Jonschkowski, R., Martín-Martín, R., Sieverling, A., Wall, V., & Brock, O. (2016). Lessons from the Amazon Picking Challenge: Four aspects of building robotic systems. In Proceedings of Robotics: Science and Systems.

  20. Espiau, B., Chaumette, F., & Rives, P. (1992). A new approach to visual servoing in robotics. IEEE Transactions on Robotics and Automation, 8(3), 313–326.

    Article  Google Scholar 

  21. Hastie, T., Tibshirani, R., Friedman, J., & Franklin, J. (2005). The elements of statistical learning: Data mining, inference and prediction. The Mathematical Intelligencer, 27(2), 83–85.

    Google Scholar 

  22. Hawes, N., Zillich, M., & Jensfelt, P. (2010). Lessons learnt from scenario-based integration. In Cognitive Systems (pp 423–438). Springer.

  23. Hernandez, C., Bharatheesha, M., Ko, W., Gaiser, H., Tan, J., van Deurzen, K., de Vries, M., Mil, B. V., van Egmond, J., Burger, R., Morariu, M., Ju, J., Gerrmann, X., Ensing, R., van Frankenhuyzen, J., & Wisse, M. (2016). Team delft’s robot winner of the Amazon Picking Challenge 2016. CoRR arXiv:1610.05514.

  24. Hollerbach, J., Khalil, W., & Gautier, M. (2008). Model identification. In B. Siciliano & O. Khatib (Eds.), Springer handbook of robotics (pp. 321–344). Berlin: Springer.

    Google Scholar 

  25. Holmberg, R., & Khatib, O. (2000). Development and control of a holonomic mobile robot for mobile manipulation tasks. The International Journal of Robotics Research, 19(11), 1066–1074.

    Article  MATH  Google Scholar 

  26. Ijspeert, A. J., Nakanishi, J., Hoffmann, H., Pastor, P., & Schaal, S. (2013). Dynamical movement primitives: Learning attractor models for motor behaviors. Neural Computation, 25(2), 328–373.

    MathSciNet  Article  MATH  Google Scholar 

  27. Johnson, M., Shrewsbury, B., Bertrand, S., Wu, T., Duran, D., Floyd, M., et al. (2015). Team IHMC’s lessons learned from the DARPA robotics challenge trials. Journal of Field Robotics, 32(2), 192–208.

    Article  Google Scholar 

  28. Jonschkowski, R., & Brock, O. (2015). Learning state representations with robotic priors. Autonomous Robots, 39(3), 407–428.

    Article  Google Scholar 

  29. Jonschkowski, R., Eppner, C., Höfer, S., Martín-Martín, R., & Brock, O. (2016). Probabilistic multi-class segmentation for the Amazon Picking Challenge. In IEEE/RSJ international conference on intelligent robots and systems (pp. 822–830).

  30. Kamiya, Y. (2016). Team preferred networks. In Talk at workshop on automation for warehouse logistics (CASE2016). Accessed 16 Feb 2016.

  31. Katz, D., & Brock, O. (2011). A factorization approach to manipulation in unstructured environments. In Robotics Research (pp. 285–300). Springer.

  32. Laumond, J. P. (2006). Kineo CAM: A success story of motion planning algorithms. IEEE Robotics & Automation Magazine, 13(2), 90–93.

    Article  Google Scholar 

  33. LaValle, S. M. (2006). Planning algorithms. Cambridge: Cambridge University Press.

    Google Scholar 

  34. Lozano-Perez, T., Mason, M. T., & Taylor, R. H. (1984). Automatic synthesis of fine-motion strategies for robots. The International Journal of Robotics Research, 3(1), 3–24.

    Article  Google Scholar 

  35. Marder-Eppstein, E., Berger, E., Foote, T., Gerkey, B., & Konolige, K. (2010). The office marathon: Robust navigation in an indoor office environment. In IEEE International Conference on Robotics and Automation (ICRA) (pp. 300–307).

  36. Marr, D. (1982). Vision: A computational approach. San Francisco: Freeman & Co.

    Google Scholar 

  37. Martín-Martín, R., & Brock, O. (2014). Online interactive perception of articulated objects with multi-level recursive estimation based on task-specific priors. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (pp. 2494–2501).

  38. Martín-Martín, R., Höfer, S., & Brock, O. (2016). An integrated approach to visual perception of articulated objects. In IEEE International Conference on Robotics and Automation (ICRA) (pp. 5091–5097).

  39. Mason, M. T., Rodriguez, A., Srinivasa, S. S., & Vazquez, A. S. (2012). Autonomous manipulation with a general-purpose simple hand. The International Journal of Robotics Research, 31(5), 688–703.

    Article  Google Scholar 

  40. Milan, A. (2016). Team NimbRo. In: Talk at Workshop on Automation for Warehouse Logistics (CASE2016). Accessed 16 Feb 2016.

  41. Miller, A. T., & Allen, P. K. (2004). GraspIt! a versatile simulator for robotic grasping. IEEE Robotics & Automation Magazine, 11(4), 110–122.

    Article  Google Scholar 

  42. Morrison, D., Tow, A. W., McTaggart, M., Smith, R., Kelly-Boxall, N., Wade-McCue, S., Erskine, J., Grinover, R., Gurman, A., Hunn, T., Lee, D., Milan, A., Pham, T., Rallos, G., Razjigaev, A., Rowntree, T., Vijay, K., Zhuang, Z., Lehnert, C. F., Reid, I. D., Corke, P., & Leitner, J. (2017). Cartman: The low-cost cartesian manipulator that won the amazon robotics challenge. CoRR arXiv:1709.06283.

  43. Nakanishi, J., Cory, R., Mistry, M., Peters, J., & Schaal, S. (2008). Operational space control: A theoretical and empirical comparison. The International Journal of Robotics Research, 27(6), 737–757.

    Article  Google Scholar 

  44. Papadimitriou, C. H., & Tsitsiklis, J. N. (1987). The complexity of Markov decision processes. Mathematics of Operations Research, 12(3), 441–450.

    MathSciNet  Article  MATH  Google Scholar 

  45. Pfeifer, R., & Gómez, G. (2009). Morphological computation: Connecting body, brain, and environment. In Creating brain-like intelligence (pp. 66–83). Springer.

  46. Quigley, M., Conley, K., Gerkey, B., Faust, J., Foote, T., Leibs, J., Wheeler, R., & Ng, A. Y. (2009). ROS: An open-source robot operating system. In ICRA workshop on open source software.

  47. Rennie, C., Shome, R., Bekris, K. E., & De Souza, A. F. (2016). A dataset for improved rgbd-based object detection and pose estimation for warehouse pick-and-place. IEEE Robotics and Automation Letters, 1(2), 1179–1185.

    Article  Google Scholar 

  48. Rickert, M., Sieverling, A., & Brock, O. (2014). Balancing exploration and exploitation in sampling-based motion planning. IEEE Transactions on Robotics, 30(6), 1305–1317.

    Article  Google Scholar 

  49. Rooks, B. (2006). The harmonious robot. Industrial Robot: An International Journal, 33(2), 125–130.

    Article  Google Scholar 

  50. Rusu, R. B., & Cousins, S. (2011). 3D is here: Point Cloud Library (PCL). In IEEE International Conference on Robotics and Automation (ICRA) (pp. 1–4).

  51. Scholz, J., Levihn, M., Isbell, C., & Wingate, D. (2014). A physics-based model prior for object-oriented MDPs. In Proceedings of the 31st International Conference on Machine Learning (ICML-14) (pp. 1089–1097).

  52. Schwaber, K. (2004). Agile project management with Scrum. Redmond: Microsoft Press.

    Google Scholar 

  53. Schwarz, M., Milan, A., Lenz, C., Munoz, A., Periyasamy, A. S., Schreiber, M., Schüller, S., & Behnke, S. (2017). NimbRo picking: Versatile part handling for warehouse automation. In IEEE International Conference on Robotics and Automation (ICRA).

  54. Schwarz, M., Milan, A., Periyasamy, A. S., & Behnke, S. (2018). RGB-D object detection and semantic segmentation for autonomous manipulation in clutter. The International Journal of Robotics Research, 37(4–5), 437–451.

  55. Stulp, F., & Sigaud, O. (2013). Robot skill learning: From reinforcement learning to evolution strategies. Paladyn, Journal of Behavioral Robotics, 4(1), 49–61.

    Article  Google Scholar 

  56. Sucan, I. I., & Chitta, S. (2016). MoveIt. Accessed 16 Nov 2017.

  57. Thrun, S., Burgard, W., & Fox, D. (2005). Probabilistic robotics. Cambridge: MIT Press.

    Google Scholar 

  58. Ulrich, K. (1995). The role of product architecture in the manufacturing firm. Research Policy, 24(3), 419–440.

    Article  Google Scholar 

  59. van Gigch, J. P. (1991). System design modeling and metamodeling. Berlin: Springer.

    Google Scholar 

  60. Wolpert, D. H. (1996). The lack of a priori distinctions between learning algorithms. Neural Computation, 8(7), 1341–1390.

    Article  Google Scholar 

  61. Wolpert, D. H., & Macready, W. G. (1997). No free lunch theorems for optimization. IEEE Transactions on Evolutionary Computation, 1(1), 67–82.

    Article  Google Scholar 

  62. XR4000. (2015). Nomadic technologies inc. XR4000. Accessed 29 Jan 2016.

  63. Yu, K. T., Fazeli, N., Chavan-Dafle, N., Taylor, O., Donlon, E., Lankenau, G. D., & Rodriguez, A. (2016). A summary of team MIT’s approach to the Amazon Picking Challenge 2015. ArXiv e-prints arXiv:1604.03639.

  64. Yu, P. (2016). Team MIT. In Talk at Workshop on Automation for Warehouse Logistics (CASE2016). Accessed 16 Feb 2016.

  65. Zeng, A., Yu, K. T., Song, S., Suo, D., Walker, E., Rodriguez, A., & Xiao, J. (2017). Multi-view self-supervised deep learning for 6d pose estimation in the Amazon Picking Challenge. In IEEE International Conference on Robotics and Automation (ICRA) (pp. 1386–1383).

Download references


We would like to thank Barrett Technology for their support and our team members Raphael Deimel, Roman Kolbert, Gabriel Le Roux, and Wolf Schaarschmidt, who helped creating the winning system.

Author information



Corresponding author

Correspondence to Arne Sieverling.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

We gratefully acknowledge the Funding provided by the Alexander von Humboldt foundation and the Federal Ministry of Education and Research (BMBF), the European Commission (SOMA Project, H2020-ICT-645599), the German Research Foundation (DFG) (Exploration Challenge, BR 2248/3-1), and the travel grant provided by Amazon Robotics.

S. Höfer: Amazon Research. All work presented in this paper has been conducted during the author’s employment at TU Berlin.

This is one of several papers published in Autonomous Robots comprising the “Special Issue on Robotics Science and Systems”.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Eppner, C., Höfer, S., Jonschkowski, R. et al. Four aspects of building robotic systems: lessons from the Amazon Picking Challenge 2015. Auton Robot 42, 1459–1475 (2018).

Download citation


  • Robotic systems
  • Amazon Picking Challenge
  • Warehouse automation
  • Mobile manipulation