Avoid Missing the Effect

Abstract

Given the amount of time, money and effort that is spent on developing new algorithms and information systems, surprisingly little time is used to teach how to conduct a valuable evaluation. When evaluation is discussed, the focus is often on avoiding Type I errors, where a nonexistent effect is incorrectly accepted as existing. In contrast, this chapter focuses on Type II errors, where an existing effect is missed.

Type II errors often receive less attention even though, especially in medicine, making such errors may affect people’s quality of life. When a new information system is built, it is intended to improve an existing medical problem. Missing an existing effect wastes time and effort of one group and may incorrectly put others off the trail, thereby stopping further research and development of a potentially promising approach. With good experimental design, the chances of this can be minimized. To avoid missing an effect, the difference between means of experimental conditions should be sufficiently large. The within-group variation should be relatively small so that the difference between groups is larger than that within groups. And a large enough number of observations should be made in each experimental condition, especially with smaller effect sizes.