Skip to main content

Advertisement

SpringerLink
Log in

Research methods for design knowledge: clarifying definitions, characteristics, and areas of confusion

  • Development Article
  • Published:
Educational technology research and development Aims and scope Submit manuscript

Abstract

In the field of educational technology and instructional design, research methods are emerging that aim to curate different forms of knowledge and insights beyond traditional research studies, or what Reigeluth and An (in Reigeluth and Carr-Chellman (eds) Instructional-design theories and models: Building a common knowledge base, Lawrence Erlbaum Associates, Mahwah, 2009) refer to as “research to prove.” As a result of evolving efforts in this area, editors of research journals in the field are receiving increased submissions employing these methods but have detected some persistent confusion among authors surrounding them. This has resulted in authors submitting articles with muddled methodologies and to outlets that may not be a fit for the work an author seeks to share. It can even be unclear whether authors intentionally employed a specific design-related method prior to reporting. In this piece, we will cover four methods—instructional design cases, case studies, design-based research, and formative evaluation of designs/products—to provide clarity for both graduate students and researchers. For each of these, we will provide definitions, discuss exemplars and features of exemplars, summarize key features that should be present in such a study and its reporting, and provide guidance on front-end intentional design and planning for research studies that employ these methodologies. Additional clarity on these methods can better support scholars and emerging scholars in their roles as researchers, authors, and reviewers.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  • Boling, E. (2004). Editor’s notes: Broadening our perspective: Instructional technology and human performance technology. TechTrends, 48(2), 3. https://doi.org/10.1007/bf02762532

    Article  Google Scholar 

  • Boling, E. (2010). The need for design cases: Disseminating design knowledge. International Journal of Designs for Learning, 1(1), 1–10.

    Article  Google Scholar 

  • Boling, E. (2020). The nature and use of precedent in designing. In J. K. McDonald & R. E. West (Eds.), Design for Learning: Principles, Processes, and Praxis. EdTech Books.

    Google Scholar 

  • Barab, S. (2014). Design-based research: A methodological toolkit for engineering change. In R. K. Sawyer (Ed.), The Cambridge handbook of the learning sciences (pp. 151–170). Cambridge University Press. https://doi.org/10.1017/CBO9781139519526.011

    Chapter  Google Scholar 

  • Barab, S., & Squire, K. (2004). Design-based research: Putting a stake in the ground. The Journal of the Learning Sciences, 13(1), 1–14. https://doi.org/10.1207/s15327809jls1301_1

    Article  Google Scholar 

  • Brown, A. L. (1992). Design experiments: Theoretical and methodological challenges in creating complex interventions in classroom settings. The Journal of the Learning Sciences, 2(2), 141–178. https://doi.org/10.1207/s15327809jls0202_2

    Article  Google Scholar 

  • Brown, A., & Green, T. D. (2006). The essentials of instructional design: Connecting fundamental principles with process and practice. Pearson.

    Google Scholar 

  • Brown, J. L., & Kiernan, N. E. (2001). Assessing the subsequent effect of a formative evaluation on a program. Evaluation and Program Planning, 24(2), 129–143. https://doi.org/10.1016/S0149-7189(01)00004-0

    Article  Google Scholar 

  • Cheng, Z., Richardson, J., & Newby, T. (2020). Using digital badges as goal-setting facilitators: A multiple case study. Journal of Computing in Higher Education, 32, 406–428.

    Article  Google Scholar 

  • Cobb, P., diSessa, A., Lehrer, R., & Schauble, L. (2003). Design experiments in educational research. Educational Researcher, 32(1), 9–13. https://doi.org/10.3102/0013189X032001009

    Article  Google Scholar 

  • Collins, A. (1992). Toward a design science of education. In E. Scanlon & T. O’Shea (Eds.), New directions in educational technology (pp. 15–22). Springer. https://doi.org/10.1007/978-3-642-77750-9_2

    Chapter  Google Scholar 

  • Collins, J. W., Easterling, J., Fountain, E. J., & Stewart, H. (2004). Impact of mobile computing on the learning environment: A case study at Seton Hall University. Journal of Computing in Higher Education, 16, 128–149.

    Article  Google Scholar 

  • Dick, W. (1980). Formative evaluation in instructional development. Journal of Instructional Development, 3(3), 3–6.

    Article  Google Scholar 

  • Dick, W. (1987). Formative evaluation: Prospects for the future. Educational Technology, 27(10), 55–57.

    Google Scholar 

  • Dick, W., & Carey, L. (1996). The systematic design of instruction (4th ed.). Longman.

    Google Scholar 

  • Dick, W., Carey, L., & Carey, J. O. (2008). The systematic design of instruction. Allyn & Bacon.

  • Easterday, M. W., Lewis, D. R., & Gerber, E. M. (2014). Design-based research process: Problems, phases, and applications. In J. L. Polman, E. A. Kyza, K. O’Neill, I. Tabak, W. R. Penuel, A. S. Jurow, K. O’Connor, T. Lee, & L. D’Amico (Eds.), Learning and becoming in practice: The International Conference of the Learning Sciences (ICLS) 2014 (Vol. 1, pp. 317–324). International Society of the Learning Sciences.

    Google Scholar 

  • Edelson, D. C. (2002). Design research: What we learn when we engage in design. The Journal of the Learning Sciences, 11(1), 105–121. https://doi.org/10.1207/S15327809JLS1101_4

    Article  Google Scholar 

  • Fitzpatrick, S., Sanders, J., & Worthen, R. (2011). Program evaluation, alternative approach, and practical guidelines. Pearson Education Inc.

    Google Scholar 

  • Flagg, B. N. (1990). Formative evaluation for educational technologies. Lawrence Erlbaum Associates.

    Google Scholar 

  • Fuchs, L. S., & Fuchs, D. (1986). Effects of systematic formative evaluation: A meta-analysis. Exceptional Children, 53(3), 199–208. https://doi.org/10.1177/001440298605300301

    Article  Google Scholar 

  • Gray, C. M. (2020). Markers of quality in design precedent. International Journal of Designs for Learning, 11(3), 1–12. https://doi.org/10.14434/ijdl.v11i3.31193

    Article  Google Scholar 

  • Hodges, C., & Fowler, D. (2021). COVID-19 crisis and faculty members in higher education: From emergency remote teaching to better teaching through reflection. International Journal of Multidisciplinary Perspectives in Higher Education, 5(1), 118–122. https://doi.org/10.32674/jimphe.v5i1.2507

    Article  Google Scholar 

  • Honebein, P. C., & Reigeluth, C. M. (2021). To prove or improve, that is the question: The resurgence of comparative, confounded research between 2010 and 2019. Educational Technology Research and Development, 69(2), 465–496. https://doi.org/10.1007/s11423-021-09988-1

    Article  Google Scholar 

  • Howard, C. D. (2011). Writing and rewriting the instructional design case: A view from two sides. International Journal of Designs for Learning, 2(1), 1–10.

    Article  Google Scholar 

  • Howard, C. D., Boling, E., Rowland, G., & Smith, K. M. (2012). Instructional design cases and why we need them. Educational Technology, 1, 34–38.

    Google Scholar 

  • Lung, M. L., & Howard, C. D. (2022). Eureka! An early career designer’s insight on the design process. The International Journal of Designs for Learning, 13(2), 40–54.

    Article  Google Scholar 

  • Luo, T., & Xie, Q. (2019). Using Twitter as a pedagogical tool in two classrooms: A comparative case study between an education and a communication class. Journal of Computing in Higher Education, 31, 81–104.

    Article  Google Scholar 

  • McKenney, S., & Reeves, T. C. (2012). Conducting educational design research (1st ed.). Routledge.

    Google Scholar 

  • McKenney, S., & Reeves, T. C. (2019). Conducting educational design research (2nd ed.). Routledge. https://doi.org/10.4324/9781315105642

    Book  Google Scholar 

  • McKenney, S., & Reeves, T. C. (2021). Educational design research: Portraying, conducting, and enhancing productive scholarship. Medical Education, 55(1), 82–92. https://doi.org/10.1111/medu.14280

    Article  Google Scholar 

  • Merriam, S. (1998). Qualitative research and case study applications in education. Jossey-Bass.

    Google Scholar 

  • Merriam, S. (2009). Qualitative research: A guide to design and implementation. Jossey-Bass.

    Google Scholar 

  • Morrison, G. R., Ross, S. M., Kalman, H. K., & Kemp, J. E. (2013). Designing effective instruction (7th ed.). Wiley.

    Google Scholar 

  • Mulcahy, R. (2011). Bottom line: Defining success in the creation of a business simulation. International Journal of Designs for Learning, 2(1), 1–17.

    Article  Google Scholar 

  • Muljana, P. S., & Luo, T. (2021). Utilizing learning analytics in course design: Voices from instructional designers in higher education. Journal of Computing in Higher Education, 33, 206–234. https://doi.org/10.1007/s12528-020-09262-y

    Article  Google Scholar 

  • Oliver, K., & Hannafin, M. (2001). Developing and refining mental models in open-ended learning environments: A case study. Educational Technology Research & Development, 49, 5–32. https://doi.org/10.1007/BF02504945

    Article  Google Scholar 

  • Patton, M. (1990). Qualitative evaluation and research methods (2nd ed.). SAGE.

    Google Scholar 

  • Penuel, W. R., Fishman, B. J., Cheng, B., & Sabelli, N. (2011). Organizing research and development at the intersection of learning, implementation, and design. Educational Researcher, 40(7), 331–337. https://doi.org/10.3102/0013189X11421826

    Article  Google Scholar 

  • Ragin, C., & Schneider, G. (2011). Case-oriented theory building and theory testing. In M. Williams & W. P. Vogt (Eds.), The SAGE Handbook of innovation in social research methods (pp. 150–166). SAGE.

    Chapter  Google Scholar 

  • Reeves, T. C., & Hedberg, J. G. (2003). Interactive learning systems evaluation. Educational Technology Publications.

    Google Scholar 

  • Reigeluth, C. M., & An, Y. (2009). Theory building. In C. M. Reigeluth & A. Carr-Chellman (Eds.), Instructional-design theories and models: Building a common knowledge base (Vol. 3, pp. 365–386). Lawrence Erlbaum Associates.

    Chapter  Google Scholar 

  • Rogers, E. (2003). Diffusion of innovations (5th ed.). Free Press.

    Google Scholar 

  • Rowland, G. (2007). Educational inquiry in transition: Research and design. Educational Technology, 47(2), 14–28.

    Google Scholar 

  • Severance, S., Penuel, W. R., Sumner, T., & Leary, H. (2016). Organizing for teacher agency in curricular co-design. Journal of the Learning Sciences, 25(4), 531–564. https://doi.org/10.1080/10508406.2016.1207541

    Article  Google Scholar 

  • Shah, J. K., Ensminger, D. C., & Thier, K. (2015). The time for design-based research is right and right now. Mid-Western Educational Researcher, 27(2), 152–171.

    Google Scholar 

  • Smith, K. (2010). Producing the rigorous design case. International Journal of Designs for Learning, 1(1), 917–980.

    Google Scholar 

  • Smith, L. (1978). An evolving logic of participant observation, educational ethnography, and other case studies. In L. Shulman (Ed.), Review of researching education (pp. 316–377). F. E. Peacock.

  • Smith, K. M., & Boling, E. (2009). What do we make of design? Design as a concept in educational technology. Educational Technology, 1, 3–17.

    Google Scholar 

  • Stake, R. (1994). Case studies. In N. K. Denzin & Y. S. Lincoln (Eds.), Handbook of qualitative research (pp. 236–247). Sage.

  • Stake, R. (1995). The art of case study research. Sage.

  • Stufflebeam, D. L., & Zhang, G. (2017). The CIPP evaluation model: How to evaluate for improvement and accountability. Guilford Publications.

    Google Scholar 

  • Sugar, W. (2014). Development and formative evaluation of multimedia case studies for instructional design and technology students. TechTrends, 58(5), 36–52. https://doi.org/10.1007/s11528-014-0785-z

    Article  Google Scholar 

  • Svihla, V. (2014). Advances in design-based research. Frontline Learning Research, 2(4), 35–45. https://doi.org/10.14786/flr.v2i4.114

    Article  Google Scholar 

  • Svihla, V., & Boling, E. (2020). Introduction to design case chapters. In M. J. Bishop, E. Boling, J. Elan, & V. Svihla (Eds.), Handbook of research in educational communications and technology. Springer.

    Google Scholar 

  • Tessmer, M. (1993). Planning and conducting formative evaluations: Improving the quality of education and training. Kogan Page.

    Google Scholar 

  • The Design-Based Research Collective. (2003). Design-based research: An emerging paradigm for educational inquiry. Educational Researcher, 32(1), 5–8. https://doi.org/10.3102/0013189X032001005

    Article  Google Scholar 

  • Vanderhoven, E., Schellens, T., Vanderlinde, R., & Valcke, M. (2016). Developing educational materials about risks on social network sites: A design based research approach. Educational Technology Research & Development, 64, 459–480. https://doi.org/10.1007/s11423-015-9415-4

    Article  Google Scholar 

  • Wang, F., & Hannafin, M. J. (2005). Design-based research and technology-enhanced learning environments. Educational Technology Research and Development, 53, 5–23. https://doi.org/10.1007/BF02504682

    Article  Google Scholar 

  • Waters, S. H., & Gibbons, A. S. (2004). Design languages, notation systems, and instructional technology: A case study. Educational Technology Research & Development, 52, 57–68. https://doi.org/10.1007/BF02504839

    Article  Google Scholar 

  • Weston, C., McAlpine, L., & Bordonaro, T. (1995). A model for understanding formative evaluation in instructional design. Educational Technology Research and Development, 43(3), 29–48. https://doi.org/10.1007/BF02300454

    Article  Google Scholar 

  • Yin, R. (2008). Case study research: Design and methods. SAGE.

    Google Scholar 

  • Yin, R. K. (2012). Case study methods. In H. Cooper, P. M. Camic, D. L. Long, A. T. Panter, D. Rindskopf, & K. J. Sher (Eds.), APA handbook of research methods in psychology, research designs: Quantitative, qualitative, neuropsychological, and biological (pp. 141–155). American Psychological Association. https://doi.org/10.1037/13620-009

    Chapter  Google Scholar 

  • Yin, R. K. (2018). Case study research and applications: Design and methods. Sage.

    Google Scholar 

  • Young, P. A. (2014). The presence of culture in learning. In M. Spector, D. Merrill, J. Elen, & M.J. Bishop (Eds.), Handbook of research on educational communications and technology (pp. 349–361). Springer.

Further Reading

  • Creswell, J. W., & Poth, C. N. (2017). Qualitative inquiry and research design: Choosing among five approaches (4th ed.). SAGE Publications.

    Google Scholar 

  • Fraefel, U. (2014). Professionalization of pre-service teachers through university-school partnerships: Partner schools for professional development: development, implementation and evaluation of cooperative learning in schools and classes. In Paper presented at the annual WERA Focal Meeting. https://doi.org/10.13140/RG.2.1.1979.5925

  • Starman, A. B. (2013). The case study as a type of qualitative research. Journal of Contemporary Educational Studies, 1, 28–43.

    Google Scholar 

  • Yazan, B. (2015). Three approaches to case study methods in education: Yin, Merriam, and Stake. The Qualitative Report, 20(2), 134–152. https://doi.org/10.46743/2160-3715/2015.2102

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of New Mexico, 1 University of New Mexico, MSC05 3020, Albuquerque, NM, 87131, USA

    Stephanie L. Moore

  2. Department of Education and Language Education, International Christian University, 3-10-2, Osawa, Mitaka-Shi, Tokyo, Japan

    Craig D. Howard

  3. Indiana University, Education 3036, 201 N. Rose Ave., Bloomington, IN, 47405, USA

    Elizabeth Boling

  4. Instructional Psychology & Technology, Brigham Young University, 150-G MCKB, Provo, UT, 84602, USA

    Heather Leary

  5. College of Education, Georgia Southern University, LTHD (8131), Statesboro, GA, 30458, USA

    Charles B. Hodges

Authors
  1. Stephanie L. Moore
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Craig D. Howard
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Elizabeth Boling
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Heather Leary
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Charles B. Hodges
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Stephanie L. Moore.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical approval

This paper does not involve research with human participants or animals. Because this research does not involve human subjects, informed consent was not necessary.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Moore, S.L., Howard, C.D., Boling, E. et al. Research methods for design knowledge: clarifying definitions, characteristics, and areas of confusion. Education Tech Research Dev (2023). https://doi.org/10.1007/s11423-023-10271-8

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11423-023-10271-8

Keywords

Search

Navigation