Evaluation

Abstract

Evaluation is a key element in the design of IT-based artifacts. A designer finds a suitable and interesting problem to solve. Then they come up with design solutions. That is followed by the actual build phase. After they have built the artifact, the next phase is evaluating for efficiency, utility, or performance.

Evaluation is a crucial component in the design science research process. The designed IT artifact is a socio-technical entity that exists within an environment (business or social) which lays out the requirements for its evaluation. Such evaluation of IT artifacts requires definition of appropriate metrics and possibly the gathering and analysis of appropriate data. IT artifacts can be evaluated in terms of functionality, completeness, consistency, accuracy, performance, reliability, usability, fit with the organization, and other relevant quality attributes (Hevner, March et al. 2004).

In this chapter, it is our goal to help the reader understand the different issues, questions, methods, and techniques that arise when one does evaluation. To present a full detailed analysis of various techniques is beyond the scope of this chapter or the book, but we hope that the reader will learn to ask the right questions, know when to apply which technique and be confident to look at the right places for more answers.

I keep six honest serving men. They taught me all I knew. Their names are What and Why and When and How and Where and Who.

– Rudyard Kipling