Abstract
The use of experiments to verify hypotheses is one of the central elements of science. In computing, experiments—most commonly an implementation tried against test data—are used for purposes such as confirming hypotheses about algorithms and systems. An experiment can verify, for example, that a system can complete a specified task, and can do so with reasonable use of resources. A tested hypothesis becomes part of scientific knowledge if it is sufficiently well described and constructed, and if it is convincingly demonstrated.
The senses deceive from time to time, and it is prudent never to trust wholly those who have deceived us even once.
Rene Descartes
A hypothesis is ... a mere trial idea, a tentative suggestion concerning the nature of things. Until it has been tested, it should not be confused with a law ... Plausibility is not a substitute for evidence, however great may be the emotional wish to believe.
E. Bright Wilson, Jr.
An Introduction to Scientific Research
Even the clearest and most perfect circumstantial evidence is likely to be at fault, after all, and therefore ought to be received with great caution.
Mark Twain
Pudd’nhead Wilson’s Calendar
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
In this discussion I generally use data in the usual sense in computing, namely as the raw material on which experiments operate. In other contexts, data is the result or output of an experiment, such as measurements gathered in a lab or from human subjects. Confusingly, computing experiments on data produce data as output. It is the output sense of the word data that is meant in the truism “we process data to obtain information, analyze information to obtain knowledge, and comprehend knowledge to obtain wisdom”.
- 2.
The existence of these issues was obscured by chaotic reporting practices. For example, in one iteration of the work she reported total numbers of positives and total numbers of errors—both positive and negative—but did not report any of the components, such as true positives, false negatives, and so on. In another iteration, instead of reporting numbers as a function of threshold, she reported the number of positives as a function of the number of false positives, so that the threshold acted as a hidden variable. In some fundamental way she had not grasped how a trend can be used to understand the underlying behaviour of the method being investigated; in this case, it would be interesting to observe change in the number of true positives as the threshold was varied.
- 3.
Is computational efficiency even well defined? Is it the number of instructions used, say, or the number of seconds each CPU core spends on the process? These are no longer equivalent. If the core is otherwise idle, due to memory access delays, is processing time relevant at all? The answers to these issues will depend on the research question.
- 4.
Sadly, the findings were that the system was unhelpful.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
Copyright information
© 2014 Springer-Verlag London
About this chapter
Cite this chapter
Zobel, J. (2014). Experimentation. In: Writing for Computer Science. Springer, London. https://doi.org/10.1007/978-1-4471-6639-9_14
Download citation
DOI: https://doi.org/10.1007/978-1-4471-6639-9_14
Published:
Publisher Name: Springer, London
Print ISBN: 978-1-4471-6638-2
Online ISBN: 978-1-4471-6639-9
eBook Packages: Computer ScienceComputer Science (R0)