Abstract
While it is often said that in order to qualify as a true science robotics should aspire to reproducible and measurable results that allow benchmarking, I argue that a focus on benchmarking will be a hindrance for progress. Several academic disciplines that have been led into pursuing only reproducible and measurable ‘scientific’ results—robotics should be careful not to fall into that trap. Results that can be benchmarked must be specific and context-dependent, but robotics targets whole complex systems independently of a specific context—so working towards progress on the technical measure risks missing that target. It would constitute aiming for the measure rather than the target: what I call ‘measure-target confusion’. The role of benchmarking in robotics shows that the more general problem to measure progress towards more intelligent machines will not be solved by technical benchmarks; we need a balanced approach with technical benchmarks, real-life testing and qualitative judgment.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Aly, A., Griffiths, S., Stramandinoli, F.: Metrics and benchmarks in human-robot interaction: recent advances in cognitive robotics. Cognitive Systems Research (2016, forthcoming). https://doi.org/10.1016/j.cogsys.2016.06.002
Amigoni, F., Bastianelli, E., Bonarini, A., Fontana, G., Hochgeschwender, N., Iocchi, L., Schiaffonati, V.: Competitions for benchmarking. IEEE Robot. Autom. Mag. 22(3), 53–61 (2016)
Antonelli, G.: Robotic research: are we applying the scientific method? Frontiers in Robotics and AI 2, 1–4 (2015). https://doi.org/10.3389/frobt.2015.00013
Bonsignorio, F., Del Pobil, A.P.: Toward replicable and measurable robotics research. IEEE Robot. Autom. Mag. 22(3), 32–35 (2015)
Bostrom, N.: Superintelligence: paths, dangers, strategies. Oxford University Press, Oxford (2014)
Campbell, D.T.: Assessing the impact of planned social change. Eval. Program Plan. 2(1), 67–90 (1979). https://doi.org/10.1016/0149-7189(79)90048-X
Dias, J., Althoefer, K., Lima, P.U.: Robot competitions: what did we learn? IEEE Robot. Autom. Mag. (1), 16–18 (2016)
EURON: Survey and inventory of current efforts in comparative robotics research. European Robotics Research Network (2008). Retrieved from http://www.robot.uji.es/EURON/en/index.htm
Gomila, A., Müller, V.C.: Challenges for artificial cognitive systems. J. Cogn. Sci. 13(4), 453–469 (2012). https://doi.org/10.17791/jcs.2012.13.4.453
Hick, D., Wouters, P., Waltman, L., de Rijcke, S., Rafois, I.: Bibliometrics: The Leiden Manifesto for research metrics. Nature 520, 429–431 (2015). https://doi.org/10.1038/520429a
Iantovics, L.B., Rotar, C., Nechita, E.: A novel robust metric for comparing the intelligence of two cooperative multiagent systems. Procedia Comput. Sci. 96, 637–644 (2016). https://doi.org/10.1016/j.procs.2016.08.245
Kant, I.: Critique of Pure Reason (N. K. Smith, Trans.) (1791). Palgrave Macmillan, London (1929)
Kurzweil, R.: The Singularity Is Near: When Humans Transcend Biology. Viking, London (2005)
Lier, F., Wachsmuth, S., Wrede, S: Modeling software systems in experimental robotics for improved reproducibility: a case study with the iCub humanoid robot. Humanoids (18–20 November 2014). http://pub.uni-bielefeld.de/luur/download?func=downloadFile&recordOId=2705677&fileOId=2705709
Madhavan, R., del Pobil, A.P., Messina, E.: Performance evaluation and benchmarking of robotic and automation systems (2010)
Müller, V.C.: Autonomous cognitive systems in real-world environments: less control, more flexibility and better interaction. Cogn. Comput. 4(3), 212–215 (2012). https://doi.org/10.1007/s12559-012-9129-4
Müller, V.C., Ayesh, A. (eds.): Revisiting turing and his test: comprehensiveness, qualia, and the real world, vol. 7/2012. AISB, Hove (2012)
Müller, V.C., Bostrom, N.: Future progress in artificial intelligence: a survey of expert opinion. In: Müller, V.C. (ed.) Fundamental Issues of Artificial Intelligence, pp. 553–570. Springer, Berlin (2016)
SPARC: Robotics 2020: multi-annual roadmap for robotics in Europe. Release B 03/12/2015 (2015). http://www.eu-robotics.net/
Acknowledgements
I am grateful to Fabio Bonsignorio and other members of the GEMSig, esp. Alan Winfield, for sustaining this discussion. Thanks to Barna Ivantovic for comments. I am grateful to Nick Bostrom for conversations about intelligence testing and measurement.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this chapter
Cite this chapter
Müller, V.C. (2020). Measuring Progress in Robotics: Benchmarking and the ‘Measure-Target Confusion’. In: Bonsignorio, F., Messina, E., del Pobil, A., Hallam, J. (eds) Metrics of Sensory Motor Coordination and Integration in Robots and Animals. Cognitive Systems Monographs, vol 36. Springer, Cham. https://doi.org/10.1007/978-3-030-14126-4_9
Download citation
DOI: https://doi.org/10.1007/978-3-030-14126-4_9
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-14124-0
Online ISBN: 978-3-030-14126-4
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)