Models and Experiments in Robotics

  • Francesco AmigoniEmail author
  • Viola Schiaffonati
Part of the Springer Handbooks book series (SHB)


This chapter surveys the practices that are being employed in experimentally assessing the special class of computational models embedded in robots. The assessment of these models is particularly challenging mainly due to the difficulty of accurately estimating and modeling the interactions between the robots and their environments, especially in the case of autonomous robots, which make decisions without continuous human supervision. The field of autonomous robotics has recognized this difficulty and launched a number of initiatives to deal with it. This chapter, after a conceptual premise and a broad introduction to the experimental issues of robotics, critically reviews these initiatives that range from taking inspiration from traditional experimental practices, to simulations, benchmarking, standards, and competitions.


Mobile Robot Robot System Autonomous Robot Real Robot Technical Artifact 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

autonomous agents and multiagent systems


artificial intelligence


good experimental methodology


  1. 1.
    M. Scheutz: Computation, philosophical issues about. In: The Encyclopedia of Cognitive Science, ed. by L. Nadel (Wiley, Hoboken 2002)Google Scholar
  2. 2.
    P. Vermaas, P. Kroes, I. van de Poel, M. Franssen, W. Houkes: A philosophy of technology. In: From Technical Artefacts to Sociotechnical Systems, (Morgan Claypool, San Rafael 2011)Google Scholar
  3. 3.
    S.O. Hansson: Experiments before science? – What science learned from technological experiments. In: The Role of Technology in Science. Philosophical Perspectives, ed. by S.O. Hansson (Springer, Dordrecht 2015)CrossRefGoogle Scholar
  4. 4.
    F. Amigoni, M. Reggiani, V. Schiaffonati: An insightful comparison between experiments in mobile robotics and in science, Auton. Robot. 27(4), 313–325 (2009)CrossRefGoogle Scholar
  5. 5.
    F. Amigoni, V. Schiaffonati, M. Verdicchio: Good experimental methodologies for autonomous robotics: From theory to practice. In: Methods and Experimental Techniques in Computer Engineering, Springer Briefs in Applied Sciences and Technology, ed. by F. Amigoni, V. Schiaffonati (Springer, Cham 2014) pp. 37–53CrossRefGoogle Scholar
  6. 6.
    A. Newell, H. Simon: Computer science as empirical inquiry: Symbols and search, Commun. ACM 19(3), 113–126 (1976)MathSciNetCrossRefGoogle Scholar
  7. 7.
    J.A. Feldman, W.R. Sutherland: Rejuvenating experimental computer science, Commun. ACM 22(9), 497–502 (1979)CrossRefGoogle Scholar
  8. 8.
    P.J. Denning: What is experimental computer science?, Commun. ACM 23(10), 543–544 (1980)CrossRefGoogle Scholar
  9. 9.
    W. Tichy: Should computer scientists experiment more?, Computer 31(5), 32–40 (1998)MathSciNetCrossRefGoogle Scholar
  10. 10.
    P. Freeman: Back to experimentation, Commun. ACM 51(1), 21–22 (2008)CrossRefGoogle Scholar
  11. 11.
    P. Denning, P. Freeman: Computing’s paradigm, Commun. ACM 52(12), 28–30 (2005)CrossRefGoogle Scholar
  12. 12.
    C. Morrison, R. Snodgrass: Computer science can use more science, Commun. ACM 54(6), 38–43 (2011)CrossRefGoogle Scholar
  13. 13.
    P.J. Denning: Is computer science science?, Commun. ACM 48(4), 27–31 (2005)MathSciNetCrossRefGoogle Scholar
  14. 14.
    D.G. Feitelson: Experimental computer science: The need for a cultural change (2006), manuscript,, last accessed August 2016
  15. 15.
    M.V. Zelkowitz, D.R. Wallace: Experimental validation in software engineering, Inf. Softw. Technol. 39(11), 735–743 (1997)CrossRefGoogle Scholar
  16. 16.
    M.V. Zelkowitz, D.R. Wallace: Experimental models for validating technology, Computer 31(5), 23–31 (1998)CrossRefGoogle Scholar
  17. 17.
    W. Harrison, V.R. Basili: Editorial, Empir. Softw. Eng. 1(1), 5–10 (1996)CrossRefGoogle Scholar
  18. 18.
    M. Barni, F. Perez-Gonzalez, P. Comesana, G. Bartoli: Putting reproducible signal processing into practice: A case study in watermarking, Proc. IEEE ICASSP 4, 1261–1264 (2007)Google Scholar
  19. 19.
    P. Vandewalle, J. Kovacevic, M. Vetterli: Reproducible research in signal processing, IEEE Signal Process. Mag. 26(3), 37–47 (2009)CrossRefGoogle Scholar
  20. 20.
    B. Mayer, M. Nordio: Empirical Software Engineering and Verification, Programming and Software Engineering, Vol. 7007 (Springer, Berlin, Heidelberg 2010)Google Scholar
  21. 21.
    N. Juristo, O. Gomez: Replication of software engineering experiments, Lect. Notes Comput. Sci. 7007, 60–88 (2012)CrossRefGoogle Scholar
  22. 22.
    M. Barni, F. Perez-Gonzalez: Pushing science into signal processing, IEEE Signal Process. Mag. 120, 119–120 (2005)Google Scholar
  23. 23.
    C. Drummond: Replicability is not reproducibility: Nor is it good science, Proc. Eval. Methods Mach. Learn. Workshop 26th Int. Conf. Mach. Learn. (2009)Google Scholar
  24. 24.
    S. Hanks, M. Pollack, P. Cohen: Benchmarks, test beds, controlled experimentation, and the design of agent architectures, AI Magazine 14(4), 17–42 (1993)Google Scholar
  25. 25.
    International Foundation of Robotics Research: International Symposium on Experimental Robotics,, last accessed August 2016
  26. 26.
    F. Bonsignorio, J. Hallam, A. del Pobil, Special Interest Group on Good Experimental Methodology: GEM Guidelines,, last accessed August 2016
  27. 27.
    F. Bonsignorio, J. Hallam, del Defining the requisites of a replicable robotics experiment, RSS2009 Workshop Good Exp. Methodol. Robot. 2009 (2009)Google Scholar
  28. 28.
    IEEE Robotics and Automation Magazine special issue on Replicable and Measurable Robotics Research (2015)Google Scholar
  29. 29.
    L. Takayama: Toward a science of robotics: Goals and standards for experimental research, Robot.: Sci. Syst. (RSS) Workshop Good Exp. Methodol. Robot., Seattle (2009)Google Scholar
  30. 30.
    M. Caccia, E. Saggini, M. Bibuli, G. Bruzzone, E. Zereik, E. Riccomagno: Towards good experimental methodologies for unmanned marine vehicles, Lect. Notes Comput. Sci. 8112, 365–372 (2013)CrossRefGoogle Scholar
  31. 31.
    F. Lier, J. Wienke, A. Nordmann, S. Wachsmuth, S. Wrede: The cognitive interaction toolkit – Improving reproducibility of robotic systems experiments, Lect. Notes Artif. Intell. 8810, 400–411 (2014)Google Scholar
  32. 32.
    A. Tanoto, J. Gomez, N. Mavridis, L. Hanyi, U. Rückert, S. Garrido: Teletesting: Path planning experimentation and benchmarking in the Teleworkbench, Proc. Eur. Conf. Mob. Robot. (ECMR) (2013) pp. 343–348Google Scholar
  33. 33.
    S. Hartmann: The world as a process: Simulations in the natural and social sciences. In: Simulation and Modeling in the Social Sciences from the Philosophy of Science Point of View, ed. by R. Hegselmann, U. Mueller, K.G. Troitzsch (Kluwer, Dordrecht 1996) pp. 77–100CrossRefGoogle Scholar
  34. 34.
    R. Frigg, S. Hartmann: Models in science. In: The Stanford Encyclopedia of Philosophy, ed. by E.N. Zalta (Stanford Univ., Stanford 2012), Google Scholar
  35. 35.
    E. Winsberg: Science in the Age of Computer Simulations (Univ. Chicago Press, Chicago, London 2010)CrossRefGoogle Scholar
  36. 36.
    E. Winsberg: Computer simulations in science. In: The Stanford Encyclopedia of Philosophy, ed. by E.N. Zalta (Stanford Univ., Stanford 2015), Google Scholar
  37. 37.
    D. Brugali, J. Broenink, T. Kroeger, B. MacDonald (Eds.): Simulation, Modeling, and Programming for Autonomous Robots, Lecture Notes in Artificial Intelligence, Vol. 8810 (Springer, Berlin, Heidelberg 2014)Google Scholar
  38. 38.
    USARSim (2010), last accessed April 2015
  39. 39.
    S. Carpin, T. Stoyanov, Y. Nevatia, M. Lewis, J. Wang: Quantitative assessments of USARSim accuracy, Proc. PerMIS (2006) pp. 111–118Google Scholar
  40. 40.
    B. Balaguer, S. Balakirsky, S. Carpin, M. Lewis, C. Scrapper: USARSim: A validated simulator for research in robotics and automation, IEEE/RSJ IROS 2008 Workshop Robot Simulators (2008),, last accessed August 2016Google Scholar
  41. 41.
    A. Howard, N. Roy: The robotics data set repository, radish (2003),, last accessed February 2015
  42. 42.
    Rawseeds: The Rawseeds Project,, last accessed March 2015
  43. 43.
    OpenSLAM: OpenSLAM,, last accessed April 2015
  44. 44.
    A. Winfield: Robot simulators and why I will probably reject your paper, Alan Winfield’s Web Log, November 30th 2014,, last accessed August 2016
  45. 45.
    J. Bown, P.S. Andrews, Y. Deeni, A. Goltsov, M. Idowu, F.A. Polack, A.T. Sampson, M. Shovman, S. Stepney: Engineering simulations for cancer systems biology, Curr. Drug Targets 13(12), 1560–1574 (2012)CrossRefGoogle Scholar
  46. 46.
    A. del Pobil: Research benchmarks V2, EURON European Robotics Network (2006), last accessed August 2016
  47. 47.
    R. Dillmann: KA 1.10, Benchmarks for robotics research, EURON European Robotics Network (2004), last accessed August 2016
  48. 48.
    RoSta: Robot Standards and Reference Architectures,, last accessed August 2016
  49. 49.
    R. Madhavan, R. Lakaemper, T. Kalmar-Nagy: Benchmarking and standardization of intelligent robotic systems, Proc. ICAR (2009) pp. 1–7Google Scholar
  50. 50.
    NIST: NIST Standard Test Methods for Response Robots,, last accessed August 2016
  51. 51.
    PerMIS, Performance Metrics for Intelligent Systems Workshop,, last accessed August 2016
  52. 52.
    J. Baltes: A benchmark suite for mobile robots, Proc. IROS 12, 1101–1106 (2000)Google Scholar
  53. 53.
    C. Sprunk, J. Roewekaemper, G. Parent, L. Spinello, G.D. Tipaldi, W. Burgard, M. Jalobeanu: An experimental protocol for benchmarking robotic indoor navigation, Proc. Int. Symp. Exp. Robot. (ISER) (2014)Google Scholar
  54. 54.
    R. Madhavan, C. Scrapper, A. Kleiner: guest editorial: Special issue on characterizing mobile robot localization and mapping, Auton. Robot. 27(4), 309–311 (2009)CrossRefGoogle Scholar
  55. 55.
    B. Calli, A. Singh, A. Walsman, S. Srinivasa, P. Abbeel, A. Dollar: The YCB object and model set,, last accessed August 2016
  56. 56.
    A. Shakhimardanov, E. Prassler: Comparative evaluation of robotic software integration systems: A case study, Proc. IROS (2007) pp. 3031–3037Google Scholar
  57. 57.
    O. Michel, F. Rohrer, Y. Bourquin: Rat’s life A cognitive robotics benchmark, Proc. Eur. Robot. Symp. (2008) pp. 223–232Google Scholar
  58. 58.
    B. Rohrer: Accelerating progress in artificial general intelligence: Choosing a benchmark for natural world interaction, J. Artif. Gen. Intell. 2(1), 1–28 (2011)MathSciNetCrossRefGoogle Scholar
  59. 59.
    M. Zillich: My robot is smarter than your robot – On the need for a total Turing test for robots, Proc. AISB/IACAP Symp. Revisiting Turing his Test Compr., Qualia, Real World (2012) pp. 12–15Google Scholar
  60. 60.
    A. Torralba, A. Efros: Unbiased look at dataset bias, Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR) (2011) pp. 1521–1528Google Scholar
  61. 61.
    G. Virk, S. Cameron: ISO-IEC standardization efforts in robotics, Proc. IROS Workshop Stand. Knowl. Represent. Ontol. Robot. Autom. (2014) pp. 5–6Google Scholar
  62. 62.
    R. Madhavan, W. Yu, C. Schlenoff, E. Prestes, F. Amigoni: Draft standards development of two working groups, IEEE Robot. Autom. Mag. 21(3), 20–23 (2014)CrossRefGoogle Scholar
  63. 63.
    OMG: Robotics Domain Task Force,, last accessed August 2016
  64. 64.
    P. Bonasso, T. Dean: A retrospective of the AAAI robot competitions, AI Magazine 18(1), 11–23 (1997)Google Scholar
  65. 65.
    R. Murphy: Using robot competitions to promote intellectual development, AI Magazine 21(1), 77–90 (2000)Google Scholar
  66. 66.
    J. Casper, M. Micire, J. Hyams, R. Murphy: A case study of how mobile robot competitions promote future research, Lect. Notes Artif. Intell. 2377, 123–132 (2002), RoboCup 2001zbMATHGoogle Scholar
  67. 67.
    M. Chew, S. Demidenko, C. Messom, G. Sen Gupta: Robotics competitions in engineering education, Proc. 4th Int. Conf. Auton. Robot. Agents (2009) pp. 624–627Google Scholar
  68. 68.
    J. Anderson, J. Baltes, C.T. Cheng: Robotics competitions as benchmarks for AI research, Knowl. Eng. Rev. 26(1), 11–17 (2011)CrossRefGoogle Scholar
  69. 69.
    T. Bräunl: Research relevance of mobile robot competitions, IEEE Robot. Autom. Mag. 6(4), 32–37 (1999)CrossRefGoogle Scholar
  70. 70.
    M. Anderson, O. Jenkins, S. Osentoski: Recasting robotics challenges as experiments, IEEE Robot. Autom. Mag. 18(2), 10–11 (2011), doi: 10.1109/MRA.2011.941627 CrossRefGoogle Scholar
  71. 71.
    S. Behnke: Robot competitions – Ideal benchmarks for robotics research, Proc. IROS Workshop Benchmarks Robot. Res. (2006),, last accessed August 2016Google Scholar
  72. 72.
    H. Yanco: Designing metrics for comparing the performance of robotic systems in robot competitions, Workshop Meas. Perform. Intel. Intel. Syst. (PERMIS) (2001)Google Scholar
  73. 73.
    E. Messina, R. Madhavan, S. Balakirsky: The role of competitions in advancing intelligent systems: A practioner’s perspective, Proc. PerMIS (2009) pp. 105–108CrossRefGoogle Scholar
  74. 74.
    J. Parker, J. Godoy, W. Groves, M. Gini: Issues with methods for scoring competitors in RoboCup rescue, AAMAS Workshop Auton. Robot. Multirobot Syst. (2014),, last accessed August 2016Google Scholar
  75. 75.
    The Robocup Foundation: RoboCup,, last accessed August 2016
  76. 76.
    L. Iocchi, D. Holz, J. Ruiz-del-Solar, K. Sugiura, T. van der Zant: RoboCup@Home: Analysis and results of evolving competitions for domestic and service robots, Artif. Intell. 229, 258–281 (2015), doi: 10.1016/j.artint.2015.08.002 MathSciNetCrossRefGoogle Scholar
  77. 77.
    DARPA: DARPA Robotics Challenge,, last accessed August 2016
  78. 78.
    euRathlon: euRathlon-an outdoor robotics challenge for land, sea and air,, last accessed August 2016
  79. 79.
    RoCKIn, RoCKIn project,, last accessed August 2016
  80. 80.
    J. Hernández-Orallo: AI evaluation: Past, present and future,, last accessed August 2016
  81. 81.
    M. Tedre: The Science of Computing: Shaping a Discipline (CRC, Boca Raton 2015)zbMATHGoogle Scholar
  82. 82.
    R.N. Giere: Explaining Science: A Cognitive Approach (Univ. Chicago Press, Chicago 1998)Google Scholar
  83. 83.
    A.M. Isaac: Modeling without representation, Synthese 190(16), 3611–3623 (2013)MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2017

Authors and Affiliations

  1. 1.Dipartimento di Elettronica, Informazione e BioingegneriaPolitecnico di MilanoMilanoItaly

Personalised recommendations