Politics of Machine Learning Evaluation

Is the data good enough for training purposes? Does the model perform accurately enough? Is the error rate low enough? Such questions of ‘good enough’ are at the very core of the process of Machine Learning (ML) evaluation and can also be considered a highly political process in the development of ML systems. There is already a growing interest in the political implications of ML in relation to, for example, dataset construction and the political capacities of specific ML models or foundational algorithmic techniques. However, there has been less focus on the politics of evaluation practices and techniques in ML. To further explore this issue, we invite contributions to a topical collection on ‘The Politics of Machine Learning Evaluation’ in Digital Society. We invite papers that engage with conceptual, methodological, and political questions in relation to topics, such as but are not limited to:

- Dataset construction

- Data labelling practices

- Ground truths and benchmarks

- Biases in evaluation

- Metrics

- Errors and error analysis

- Evaluation techniques

Concretely, we invite papers that engage in conceptualising or historizing machine-learning evaluation as a politically contested practice. In addition, we are interested in papers that provide methodological approaches to the study of evaluation techniques or empirical studies into ML evaluation in practice.

This topical collection emerged as part of a workshop hosted by the University of Amsterdam in November 2023 and will feature articles presented during the event, but we also welcome additional contributions to the topic.

Timeline and submission details:

Abstracts should be between 300-500 words, excluding references. Abstracts should be sent to a.s.hansen@uva.nl and d.luitse@uva.nl, with the subject line ‘CFP: Politics of Machine Learning Evaluation’. The deadline for submission of abstracts is April 26, 2024. Notifications of invitations to submit a full paper will be sent mid-May.

Final papers are to be submitted via Digital Society’s submission system, which will be open for submission between October 18 to November 1, 2024. Please indicate that the submission is part of the topical collection.

Although initially accepted, all submissions will be subject peer review, in accordance with the peer-review procedure of Digital Society. We expect submitting authors to be the reviewer for a different paper in the collection.

If authors miss the deadline for abstract submission, they should still contact the guest editors before sending their manuscript to DISO.