Abstract
Evaluation is a central and essential activity in conducting rigorous Design Science Research (DSR), yet there is surprisingly little guidance about designing the DSR evaluation activity beyond suggesting possible methods that could be used for evaluation. This paper extends the notable exception of the existing framework of Pries-Heje et al [11] to address this problem. The paper proposes an extended DSR evaluation framework together with a DSR evaluation design method that can guide DSR researchers in choosing an appropriate strategy for evaluation of the design artifacts and design theories that form the output from DSR. The extended DSR evaluation framework asks the DSR researcher to consider (as input to the choice of the DSR evaluation strategy) contextual factors of goals, conditions, and constraints on the DSR evaluation, e.g. the type and level of desired rigor, the type of artifact, the need to support formative development of the designed artifacts, the properties of the artifact to be evaluated, and the constraints on resources available, such as time, labor, facilities, expertise, and access to research subjects. The framework and method support matching these in the first instance to one or more DSR evaluation strategies, including the choice of ex ante (prior to artifact construction) versus ex post evaluation (after artifact construction) and naturalistic (e.g., field setting) versus artificial evaluation (e.g., laboratory setting). Based on the recommended evaluation strategy(ies), guidance is provided concerning what methodologies might be appropriate within the chosen strategy(ies).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Checkland, P., Scholes, J.: Soft Systems Methodology in Practice. J. Wiley, Chichester (1990)
Cleven, A., Gubler, P., Hüner, K.: Design Alternatives for the Evaluation of Design Science Research Artifacts. In: Proceedings of the 4th International Conference on Design Science Research in Information Systems and Technology (DESRIST 2009). ACM Press, Malvern (2009)
Gregor, S., Jones, D.: The Anatomy of a Design Theory. Journal of the Association for Information Systems 8, 312–335 (2007)
Gummesson, E.: Qualitative Methods in Management Research. Studentlitterature, Chartwell-Bratt, Lund, Sweden (1988)
Hevner, A.R., March, S.T., Park, J., Ram, S.: Design Science In Information Systems Research. MIS Quarterly 28, 75–105 (2004)
March, S.T., Smith, G.F.: Design and natural science research on information technology. Decision Support Systems 15, 251–266 (1995)
Markus, M.L., Majchrzak, A., Gasser, L.: A design theory for systems that support emergent knowledge processes. MIS Quarterly 26, 179–212 (2002)
Nunamaker, J.F., Chen, M., Purdin, T.D.M.: Systems Development in Information Systems Research. Journal of Management Information Systems 7, 89–106 (1990/1991)
Peffers, K., Tuunanen, T., Rothenberger, M.A., Chatterjee, S.: A Design Science Research Methodology for Information Systems Research. Journal of Management Information Systems 24 (2008)
Pries-Heje, J., Baskerville, R.: The Design Theory Nexus. MIS Quarterly 32, 731–755 (2008)
Pries-Heje, J., Baskerville, R., Venable, J.R.: Strategies for Design Science Research Evaluation. In: Proceedigns of the 16th European Conference on Information Systems (ECIS 2008), Galway, Ireland (2008)
Sein, M.K., Henfridsson, O., Purao, S., Rossi, M., Lindgren, R.: Action Design Research. MIS Quarterly 35, 37–56 (2011)
Stufflebeam, D.L.: The Methodology of Metaevaluation as Reflected in Metaevaluations by the Western Michigan University Evaluation Center, pp. 95–125 (2000)
Sun, Y., Kantor, P.B.: Cross-Evaluation: A new model for information system evaluation. Journal of the American Society for Information Science and Technology 57, 614–628 (2006)
Vaishnavi, V., Kuechler, W.: Design Research in Information Systems. AISWorld (2004), http://desrist.org/design-research-in-information-systems/ (accessed March 3, 2012)
van Aken, J.E.: Management Research Based on the Paradigm of the Design Sciences: The Quest for Field-Tested and Grounded Technological Rules. Journal of Management Studies 41, 219–246 (2004)
Venable, J.R.: A Framework for Design Science Research Activities. In: Proceedings of the 2006 Information Resource Management Association Conference, Washington, DC, USA (2006)
Venable, J.R.: The Role of Theory and Theorising in Design Science Research. In: Hevner, A.R., Chatterjee, S. (eds.) Proceedings of the 1st International Conference on Design Science Research in Information Systems and Technology (DESRIST 2006), Claremont, CA, USA (2006)
Venable, J.R.: Identifying and Addressing Stakeholder Interests in Design Science Research: An Analysis Using Critical Systems Heuristics. In: Dhillon, G., Stahl, B.C., Baskerville, R. (eds.) CreativeSME 2009. IFIP AICT, vol. 301, pp. 93–112. Springer, Heidelberg (2009)
Walls, J.G., Widmeyer, G.R., El Sawy, O.A.: Building an information system design theory for vigilant EIS. Information Systems Research 3, 36–59 (1992)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2012 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Venable, J., Pries-Heje, J., Baskerville, R. (2012). A Comprehensive Framework for Evaluation in Design Science Research. In: Peffers, K., Rothenberger, M., Kuechler, B. (eds) Design Science Research in Information Systems. Advances in Theory and Practice. DESRIST 2012. Lecture Notes in Computer Science, vol 7286. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-29863-9_31
Download citation
DOI: https://doi.org/10.1007/978-3-642-29863-9_31
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-29862-2
Online ISBN: 978-3-642-29863-9
eBook Packages: Computer ScienceComputer Science (R0)