Software-Embedded Evaluation Support in Design Science Research

  • Jonas Sjöström
  • Leona Chandra Kruse
  • Amir Haj-Bolouri
  • Per Flensburg
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10844)


Even though the practice of integrating evaluative features into software has long been applied in commercially available software, it is still underrepresented in the Information Systems (IS) community. This paper presents a framework for embedded evaluation support. We are aware of the challenges of evaluation of socio-technical systems and take this issue into consideration in our framework. Our framework is the result of conceptualizations drawing from the evaluation discourse discussion on the topics of artifact evaluation in DSR. We also demonstrate our ideas through two examples of embedded evaluation support mechanisms designed and used in a DSR project in the Swedish healthcare sector.


Embedded evaluation Design science research Design guidelines 


  1. 1.
    Venable, J., Pries-Heje, J., Baskerville, R.: FEDS: a framework for evaluation in design science research. Eur. J. Inf. Syst. 25, 77–89 (2016)CrossRefGoogle Scholar
  2. 2.
    Hevner, A.R., March, S.T., Park, J., Ram, S.: Design science in information systems research. MIS Q. 28, 75–105 (2004)CrossRefGoogle Scholar
  3. 3.
    Le Goues, C., Nguyen, T., Forrest, S., Weimer, W.: GenProg: a generic method for automatic software repair. IEEE Trans. Softw. Eng. 38, 54–72 (2012)CrossRefGoogle Scholar
  4. 4.
    Le Goues, C., Forrest, S., Weimer, W.: Current challenges in automatic software repair. Softw. Qual. J. 21, 421–443 (2013)CrossRefGoogle Scholar
  5. 5.
    Sidiroglou, S., Laadan, O., Perez, C., Viennot, N., Nieh, J., Keromytis, A.D.: ASSURE: automatic software self-healing using rescue points. ACM SIGARCH Comput. Archit. News. 37, 37–48 (2009)CrossRefGoogle Scholar
  6. 6.
    Dave, R.H.: A built-in system of evaluation for reform projects and programmes in education. Int. Rev. Educ. 26, 475–482 (1980)CrossRefGoogle Scholar
  7. 7.
    Jervan, G., Peng, Z., Ubar, R., Kruus, H.: A hybrid BIST architecture and its optimization for SoC testing. In: Proceedings of the International Symposium on Quality Electronic Design 2002, pp. 273–279 (2002)Google Scholar
  8. 8.
    Nagle, H.T., Roy, S.C., Hawkins, C.F., McNamer, M.G., Fritzemeier, R.R.: Design for testability and built-in self test: a review. IEEE Trans. Ind. Electron. 36, 129–140 (1989)CrossRefGoogle Scholar
  9. 9.
    Agrawal, V.D., Kime, C.R., Saluja, K.K.: A tutorial on built-in self-test. I. Principles. IEEE Des. Test Comput. 10, 73–82 (1993)CrossRefGoogle Scholar
  10. 10.
    Agrawal, V.D., Kime, C.R., Saluja, K.K.: A tutorial on built-in self-test. 2. Applications. IEEE Des. Test Comput. 10, 69–77 (1993)CrossRefGoogle Scholar
  11. 11.
    Simm, W., Ferrario, M.A., Gradinar, A., Whittle, J.: Prototyping ‘clasp’: implications for designing digital technology for and with adults with autism. In: Proceedings of the 2014 Conference on Designing Interactive Systems, pp. 345–354 (2014)Google Scholar
  12. 12.
    Atterer, R., Wnuk, M., Schmidt, A.: Knowing the user’s every move: user activity tracking for website usability evaluation and implicit interaction. In: Proceedings of the 15th International Conference on World Wide Web, pp. 203–212 (2006)Google Scholar
  13. 13.
    Van Gemert-Pijnen, J.E.W.C., Kelders, S.M., Bohlmeijer, E.T.: Understanding the usage of content in a mental health intervention for depression: an analysis of log data. J. Med. Internet Res. 16 (2014)Google Scholar
  14. 14.
    Remenyi, D., Sherwood-Smith, M.: Maximise information systems value by continuous participative evaluation. Logist. Inf. Manag. 12, 14–31 (1999)CrossRefGoogle Scholar
  15. 15.
    Stufflebeam, D.L.: The CIPP model for evaluation. In: Kellaghan, T., Stufflebeam, D.L. (eds.) International Handbook of Educational Evaluation. Kluwer International Handbooks of Education, vol. 9, pp. 31–62. Springer, Heidelberg (2003). Scholar
  16. 16.
    Smithson, S., Hirschheim, R.: Analysing information systems evaluation: another look at an old problem. Eur. J. Inf. Syst. 7, 158–174 (1998)CrossRefGoogle Scholar
  17. 17.
    Venable, J., Pries-Heje, J., Baskerville, R.: A comprehensive framework for evaluation in design science research. In: Peffers, K., Rothenberger, M., Kuechler, B. (eds.) DESRIST 2012. LNCS, vol. 7286, pp. 423–438. Springer, Heidelberg (2012). Scholar
  18. 18.
    Sein, M., Henfridsson, O., Purao, S., Rossi, M., Lindgren, R.: Action design research. MIS Q. 35, 37–56 (2011)CrossRefGoogle Scholar
  19. 19.
    Cronholm, S., Goldkuhl, G.: Strategies for information systems evaluation-six generic types. Electron. J. Inf. Syst. Eval. 6, 65–74 (2003)Google Scholar
  20. 20.
    Bunge, M.: Philosophy of Science: From Explanation to Justification, vol. 2. Transaction Publishers, New Brunswick (1998)Google Scholar
  21. 21.
    Ågerfalk, P.J., Sjöström, J.: Sowing the seeds of self: a socio-pragmatic penetration of the web artefact. In: ACM International Conference Proceeding Series (2007)Google Scholar
  22. 22.
    Grönqvist, H., Olsson, E.M.G., Johansson, B., Held, C., Sjöström, J., Norberg, A.L., Hovén, E., Sanderman, R., van Achterberg, T., von Essen, L.: Fifteen challenges in establishing a multidisciplinary research program on eHealth research in a university setting: a case study. J. Med. Internet Res. 19, e173 (2017)CrossRefGoogle Scholar
  23. 23.
    Sjöström, J., von Essen, L., Grönqvist, H.: The origin and impact of ideals in eHealth research: experiences from the U-CARE research environment. JMIR Res. Protoc. 3 (2014)CrossRefGoogle Scholar
  24. 24.
    Sjöström, J., Ågerfalk, P.J., Hevner, A.R.: Scrutinizing privacy and accountability in online psychosocial care. IT Prof. 19, 45–51 (2017)CrossRefGoogle Scholar
  25. 25.
    Sjöström, J., Hermelin, M.: In-place translation in information systems development. In: Helfert, M., Donnellan, B. (eds.) EDSS 2012. CCIS, vol. 388, pp. 88–98. Springer, Cham (2013). Scholar
  26. 26.
    Sjöström, J., Alfonsson, S.: Supporting the therapist in online therapy. In: ECIS 2012 Proceedings, Barcelona, Spain (2012)Google Scholar
  27. 27.
    Mustafa, M.I., Sjöström, J.: Design principles for research data export: lessons learned in e-health design research. In: vom Brocke, J., Hekkala, R., Ram, S., Rossi, M. (eds.) DESRIST 2013. LNCS, vol. 7939, pp. 34–49. Springer, Heidelberg (2013). Scholar
  28. 28.
    Sjöström, J., Rahman, M.H., Rafiq, A., Lochan, R., Ågerfalk, P.J.: Respondent behavior logging: an opportunity for online survey design. In: vom Brocke, J., Hekkala, R., Ram, S., Rossi, M. (eds.) DESRIST 2013. LNCS, vol. 7939, pp. 511–518. Springer, Heidelberg (2013). Scholar
  29. 29.
    Iivari, J.: Distinguishing and contrasting two strategies for design science research. Eur. J. Inf. Syst. 24(1), 107–115 (2015). Nature Publishing GroupCrossRefGoogle Scholar
  30. 30.
    Purao, S., Baldwin, C.I., Hevner, A.R., Storey, V.C., Pries-Heje, J., Smith, B., Zhu, Y.: The Sciences of Design: Observations on an Emerging Field. Working Paper 09-056. Harvard Business School (2008)Google Scholar
  31. 31.
    Baskerville, R.L., Kaul, M., Storey, V.C.: Genres of inquiry in design science research: justification and evaluation of knowledge production. MIS Quart. 39(3), 541–564 (2015) CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • Jonas Sjöström
    • 1
  • Leona Chandra Kruse
    • 2
  • Amir Haj-Bolouri
    • 3
  • Per Flensburg
    • 3
  1. 1.Department of Informatics and MediaUppsala UniversityUppsalaSweden
  2. 2.Institute of Information SystemsUniversity of Liechtenstein‎VaduzLiechtenstein
  3. 3.Division of InformaticsUniversity WestTrollhättanSweden

Personalised recommendations