Abstract
This qualitative multi-case study explores the exercise and development of the design judgment of eight instructional design (ID) students working on design projects over one semester in graduate programs at four different institutions in the USA. Their design processes were explored through interviews and their design documents using the concepts of design judgment as reported by Nelson and Stolterman (The design way: Intentional change in an unpredictable world, MIT Press, Cambridge, 2012) and reflection-in-action as reported by Schön (The reflective practitioner: How professionals think in action, Basic Books, New York, 1983) to characterize the exercise and development of their design judgment across the semester. Findings reveal that these ID students make multiple design judgments concurrently and constantly. Their prior design experience, the course requirements and their design practice in class influence both the exercise and development of their design judgment. Lastly, the development of their design judgment is both perceptible and personal. In light of these findings, recommendations are made for how to support the development of ID students’ design judgment.
Similar content being viewed by others
Introduction
Design is a complex and unique process (Lawson, 2006; Lawson & Dorst, 2009; Stolterman, 2008) that is personal to a designer and unique to a design situation (Nelson & Stolterman, 2012), including those in which designers work on teams. It requires both unique ways of thinking (Batchelder, 1914; Cross, 2001; Nelson & Stolterman, 2012; Simon, 1988) and making judgments throughout a process that is imbued with designers’ values, beliefs, and experiential knowledge (Dorst, 2003; Nelson & Stolterman, 2012; Schön, 1983). As part of this complex process, designers repeatedly engage in tackling problems and creating solutions into which they incorporate their design perspectives.
What we do in instructional design (ID) is design, a point argued convincingly over 30 years ago based on demonstrating that instructional designers are engaged in design activities (Murphy, 1992; Rowland, 1992, 1993). However, in textbooks and definitions of the field, ID is considered as a systematic problem-solving process (Smith & Boling, 2009) which is based largely on design models representing process (Gibbons et al., 2014) and on investigation of design practice primarily through the use of existing instructional models and principles (Kirschner et al., 2002; Reigeluth et al., 2016; Wedman & Tessmer, 1993; Winer & Vázquez-Abad, 1995). In the field of ID, the prevalence of such models and principles has stressed the notion of the ID process as being linear and sequential, and separating designers from design (Smith & Boling, 2009).
Researchers have argued that instructional designers do not follow the steps provided by ID models (Kenny et al., 2004; Rowland, 1992; Smith & Boling, 2009), but they follow a cyclical decision-making process influenced by personal beliefs (Jonassen, 2008) using tools, including theoretical tools, selecting according to their instrumental judgment of design situations (Lachheb & Boling, 2018). Further, instructional designers make decisions based on formal and experiential knowledge, philosophical perspectives, and interactions with peers (Christensen & Osguthorpe, 2004). Tracey & Baaki (2014) have argued that through the practice of ID designers are reflecting their own knowledge, experiences, and backgrounds and making decisions based on these reflections. Other researchers have affirmed that designers make evaluations and decisions continuously during a design project based on their practical experience and judgment (Boling et al., 2017a; Williams et al., 2011). All have stressed that designers involve their core values and beliefs in their design practices and incorporate them into their designs by using them in conjunction with their professional knowledge. Similarly, Yanchar & Gabbitas (2011) have argued that designers use their conceptual design sense in the design process. They defined this notion as designers’ critical flexibility, which “calls for a critical stance toward one’s own design sense and an awareness of alternative views” (p. 388). They suggested that using design sense, which encompasses the ability to utilize conceptual resources, critically examine their practices, and acknowledge what they learn, would challenge the acceptance of traditional views on ID practice. In each of these emergent views of ID and ID processes, the role of design judgment becomes more critical, not less, and is therefore an important subject for study.
Referring to Kerr’s (1983) “black box” of instructional designers’ decisions, Schatz (2003) proposed that paying attention to design decisions and the evolution of design is necessary for the development of the field. In the last two decades, a shift has been identified in the field toward design (Demiral-Uzan, 2016). Attention to design judgment in ID has rapidly increased in the last decade (e.g. Boling et al., 2017a; Demiral-Uzan, 2015; Farmer & Koehler, 2022; Gray et al., 2015; Parsons et al., 2020; Zhu et al., 2020). Very recently, the field has been discussing and defining learning experience design (LXD), which stresses the use of design methods informed by user experience design (Abbott, 2020; Chang & Kuwata, 2020; Jahnke et al., 2020; Korkmaz, 2018; Schmidt & Huang, 2022; Stefaniak & Sentz, 2020; Vann & Tawfik, 2020). While the language of a broader approach to design appears in descriptions of LXD, it is important to note that design judgment is still required in order to assess and determine which methods to apply and when.
Even though studies focusing on design judgment in design practice have increased in recent years, there is still some lack of research in this area. While some studies focus on certain kind of design judgment, others explore design judgment in design practice. For instance, Boling et al. (2017a) explored the core judgment of ID practitioners and identified that those instructional designers hold core judgments, and that those judgments are clustered even though this notion has rarely been discussed in the field. In exploring how instrumental judgment has taken place in a cross-cultural user research design, Gray & Boling (2018) identified instrumental judgment being used when handling cultural dimensions of the co-design process. Lachheb & Boling (2018) have argued that instrumental judgment has an essential role in design practice where instructional designers select which tools to use, and why and have also explained how designers employ design judgment in general (Lachheb & Boling, 2020). In more specific aspects, Parsons et al. (2020) investigated how each kind of design judgment takes place in data visualization practice, while Farmer & Koehler (2022) explained design judgments taking place in designing e-learning modules. Zhu et al. (2020) explored novice instructional designers’ design practices and judgments and identified that those novice instructional designers used framing, core, instrumental, navigational and appearance judgments in their design practice. All of these scholars’ work highlights that the field of ID has paid attention to Smith’s (2008) call to think more about designers and their design judgment. More recently, in reviewing the literature on how instructional designers are educated and how expertise is developed in instructional designers, Stefaniak & Hwang (2021) identified a need to explore ID students’ design practice, decision-making, design judgment and how it is developed over time.
The purpose of this study, therefore, is to explore ID students’ exercise and development of design judgment over one semester when working on design projects in master’s level introductory ID and development courses across four institutions of higher education. By shedding light on students’ design practices as well as their exercise and development of design judgment, this investigation provides opportunities for reflecting on existing ID education and extends discussions on how to improve the preparation of instructional designers.
Conceptual framework
Design judgment
Judgment is a responsible choice, in other words a personal decision made with a sense of commitment to discovering the “rules of rightness” ‘that are applicable to a particular situation (Vickers, 1984). Nelson and Stolterman (2012) asserted that judgment is independent of rational decision-making; rather it is dependent on the accumulation of experiences of the consequences of choices in various situations. They have argued that judgment is based on knowledge that cannot be separated from the knower: it can be revealed only through actions. They see judgment as a convergent process that brings diversity and divergence into focus in complex, real-world situations. Judgment is essential in design and is used throughout the design process (Nelson & Stolterman, 2003, 2012). The construct of design judgment, as with the full range of constructs in Nelson & Stolterman (2012), is drawn from multiple bases in research and theory on judgment and decision-making (e.g. Holt (1997), Vickers (1984), and Polanyi (1966)). Nelson and Stolterman (2012) conceptualized judgment in design practice as “the ability to gain subconscious insights that have been abstracted from experiences and reflections, informed by situations that are complex, indeterminate, indefinable, and paradoxical” (p. 145). Based on this perspective, they explained design judgment comprehensively and identified multiple forms of judgment (framing, deliberated off hand, default, core, instrumental, appearance, appreciative, connective, compositional, navigational, and quality) that are exercised in various design situations.
Researchers in the field of ID have operationalized the theoretical notion of design judgments in ID practice. In exploring the design judgment of ID students (Demiral-Uzan, 2015) and instructional designers (Gray et al., 2015), it has been found that IDs continuously make design judgments while designing instruction. Design judgments are clustered in the process, rather than being exercised in isolation or serially, and they are shaped by environmental factors (Gray et al., 2015). As the abilities of ID students progress, their intensity in making design judgment increases, and the types and frequencies of their design judgments change over time (Demiral-Uzan, 2015).
Design judgment in the context of ID is defined as the ability to use foundational and experiential knowledge gained by reflecting on designing experiences over time, to combine such acquired knowledge with the designer’s own values and core perspectives, and to employ these in the design process. For the purpose of this study, operationalization of Nelson and Stolterman’s (2012) types of designer judgment has been employed as a conceptual framework to identify ID students’ design judgments made in an introductory ID course (see Table 1).
Development of design judgment
The ability to make good design judgments is learned and developed through practice over time (Holt, 1997) and through accumulating as many design experiences as possible (Nelson & Stolterman, 2012). Because design judgment is unique to each designer, individuals’ ability to make design judgments, and the development of design judgment will also vary from person to person (Holt, 1997; Nelson & Stolterman, 2012). Holt (1997) claimed that good judgment comes with preparation, reflection, maturity, and, most importantly, awareness. Nelson & Stolterman (2012) also asserted that designers can reflect on their judgment and their ability to make good judgments can improve. Other authors agree that as designers’ thought processes and ways of designing change with experience over time (Ertmer et al., 2008; Fortney & Yamagata-Lynch, 2013; Perez & Emery, 1995; Rowland, 1992), designers’ use of judgment in design develops and deepens.
Schön (1983) indicated that reflection is dependent on experience, and is the act of making evaluations drawn from experience. He observed that reflections can be made while designing (reflection- in-action), and that these are key to development of design judgment. Therefore, looking at designers’ judgment through the lens of their reflections may provide insights into the development of designers’ design judgment, and for this reason Schön’s notion of reflection-in-action has been employed in this study to explore development of students’ design judgment.
Research questions
The following research questions have therefore guided this study:
-
1.
How do ID students exercise design judgment when working on real-world design projects during master’s level introductory ID and development courses?
-
2.
How is design judgment developed in these ID students when working on real-world design projects during master’s level introductory ID and development courses?
Method
A qualitative multi-case study design was employed for this study (Hancock & Algozzine, 2006; Yin, 2014) with each ID student participant was considered as a case. Analysis and conclusions have been drawn from all the cases without the aim of comparing each one to the others.
Context
Four ID courses at four different U.S. institutions, in which students were asked to work on design projects over the course of a semester, constituted the context of the study. The selection of the sites was made with criterion and convenience sampling (Miles & Huberman, 1994; Onwuegbuzie & Leech, 2007). The sites were selected based on two criteria: (1) the ID program offers face-to-face or online master’s level introductory ID courses in the USA, and (2) students in an introductory ID course are required to design instruction or instructional materials. The final sites were selected further for the ease of access to data required for the study.
Four course instructors who were willing to participate shared their course syllabi and materials, to be used as background context for the study, and announced the study to their students. The delivery methods and duration of these courses varied as well as the structure of the design projects required of students. While some of these ID courses took a traditional approach where design project deliverables were aligned with the steps in ID models, others took a more holistic design approach that required students to write general design reports not tied to a specific process. Selecting ID courses utilizing different approaches enabled increase in the variance between cases and exploration of ID students’ design judgment development within different course structures. A summary of these contexts (Table 2) is presented in the following table.
Participants
When selecting participants of the study, a convenience sampling strategy was employed (Miles et al., 2014). Once the instructors had informed students about the study, students were asked to send an email to the researcher to indicate their consent to participate. Students who agreed to participate indicated willingness to share their design projects and to be interviewed twice during the semester about their design projects.
A total of eight ID students from four different institutions were willing to participate in the study. Students with various backgrounds (see Table 3) provided increased discrepancy between cases and explore design judgment development of ID students with various backgrounds.
Data sources and data collection
The data sources used in this study were documents, including course syllabi and materials, students’ design project deliverables and artifacts and interviews conducted with students in the middle and at the end of the semester. Course syllabi and materials were used to understand the context of the course where ID students were required to work on a design project and not included directly in the analysis for the study.
ID students’ design documents were used to understand the context of design in which their design judgments were based and exercised. These documents helped to identify possible moments in which design judgment was employed and to structure individualized interview questions for the interviews. The design documents were also used as prompts and aids during interviews when the students were talking about the design judgments they made. Students shared these documents, as they became available, via email.
Two semi-structured interviews with each student were the primary data source of the study for generating findings and themes. Interviews were scheduled with the students after their design documents were collected, design judgment moments had been identified and individualized interview questions were prepared. Each participating student was interviewed twice. In the interviews, students were asked about the design process they had used, and the design judgments that had been identified from their documents. Students were asked to share how they came to the point of making each judgment and where that idea had come from. During the interviews, students were not asked directly whether they employed certain kinds of design judgment due to the fact that they might not know the constructs employed in this study. Instead, they were asked about their thinking process when coming to moments of judgment. Each interview took about 60 min and was audio recorded with the permission of students. Two interviews with each student resulted in a total of 16 h of recorded interview data. All but three interviews were conducted online via an audio communication platform and three interviews were conducted face-to-face. During interview sessions, the researcher provided explanations regarding which design judgment moments she was referring to and for consistency in data collection during face-to-face interviews, the researcher took same approach as that of online interviews.
Data analysis
Data in the form of transcripts and students’ design documents were analyzed according to the phases of Braun & Clarke’s (2006) thematic analysis method. Through an iterative and interpretive process of reviewing students’ design documents and the interview transcripts, coding was carried out based on conceptualization of types of design judgment (Nelson & Stolterman, 2012), operationalization of design judgments (Demiral-Uzan, 2015) and the notion of reflection-in-action (Schön, 1983). Qualitative data was supported by the use of NVivo 11 qualitative data analysis software.
A step-by-step approach was taken when analyzing the data. Each student’s design project deliverables were analyzed separately before conducting an interview with that student. For each design project deliverable, the types of design judgment based on Nelson & Stolterman’s (2012) conceptualization of types of design judgment were coded. This coding helped to map the students’ design judgment and to prepare individualized interview questions.
Once the interview data were transcribed, the transcripts were read through to check for any errors and re-read to be familiar with the data. When analyzing the interview data, Demiral-Uzan’s (2015) operationalization of each kind of design judgment was employed and this framework was expanded as new actions were identified and associated to a type of design judgment in the framework. An extended framework was used in analyzing students’ interview data (Table 1).
Table 4 demonstrates by example how the interview data was coded in terms of types of design judgments. This excerpt is from Jian’s second interview where she talked about how she decided to use a selected story in her design. This excerpt also demonstrates how different types of design judgments are clustered in a design moment.
After coding was completed in terms of types of design judgment to explore students’ exercise of design judgment, each segment was coded again based on Schön’s (1983) notion of reflection-in-action to determine whether reflection was employed or not when making these judgments. Indicators used for this coding were: whether the consequences of choices were considered; how the moves that resulted in the actual design were tried out; and whether inquiry was involved in the design situation. For instance, when students deeply considered and questioned what they had done, how their choice played out in design, why they made a choice and which path they needed to take in design, these segments were coded as design judgment made with reflection. Statements not showing these indicators were considered to demonstrate lack of reflection in making that design judgment. This helped to explore how students’ design judgment was developing based on whether they were making design judgments with reflection or not.
After two forms of coding were completed, descriptions of each case in terms of students’ design judgment exercise and development were generated. These descriptions helped to move to an abstract level in analyzing the data and generating themes in terms of how these students exercised their design judgment and how their design judgment developed over the semester.
Trustworthiness and transferability
Triangulation (Creswell & Miller, 2000; Merriam, 2009; Patton, 2015) and reflexive journaling (Creswell & Miller, 2000; Lincoln & Guba, 2003; Merriam, 2009) were used to establish trustworthiness. Triangulation was employed by using multiple sources of data (students’ documents and transcripts of their interviews), and using multiple investigators when analyzing the data. A critical researcher review and expert review were sought (Patton, 2015). While coding the data, the researcher met on a regular basis with a colleague who has experience in qualitative research but was not familiar with the concepts of design judgment. He first familiarized himself with the framework used in the study, and coded an excerpt of data using the framework. Subsequently, we met to compare our codes and discuss the reason for coding certain kinds of design judgments. Later, the final report of the findings was shared with him; he reviewed the report critically and asked probing questions about the findings. In addition to this critical researcher review, the co-authors regularly met when analyzing the data and discussed the analysis process. Moreover, the second author, who has expertise in design pedagogy and judgment, reviewed the report of the findings, asking critical questions about coding, interpretations and ways of reporting. In both cases, efforts to resolve questions and reconcile understandings led to increased confidence in the analysis. Furthermore, recruiting participants from multiple ID courses at multiple institutions also helped with triangulating across different contexts and multiple sources (students’ design artifacts and interviews with them) (Gerring, 2007).
Because of the nature of design, and because design judgments are unique to designers, this study did not aim to generalize the findings across ID students either in the same course or across ID courses from different programs. The aim of the study is rather to transfer the implicit meaning embedded in the phenomenon (ID students’ design judgment development) to explicit discussions of the phenomenon in order to construct shared understanding of the concept of design judgment development in the field.
Findings
In this section, each ID students’ exercise of design judgment and their development regarding judgments made with or without reflection are presented.
Jian’s design judgment exercise and development
For her ID project, Jian designed instruction for second and third grade Chinese students, a population and topic she is familiar with, intending to increasing their reading comprehension. Table 5 summarizes Jian’s design judgments throughout her design process and how her design judgment developed over time.
Jian’s use of reflection in making design judgments fluctuated. In the early stage of framing her design, although she employed reflection in some cases, she did not in others. In some cases, her accumulated experience of teaching enabled her to reflect while making design judgments. In other cases, her experience caused her to make judgments without reflection. That might be because she had already reflected on her judgments before she began her design process and thus knew what she would do. In regard to planning her design, she seemed to make judgments without reflection. However, when she was developing her instruction, she revisited her judgments and employed reflection. At the end of the design project, when she was conducting a formative evaluation, she did not employ reflection but instead tended to follow guidelines. Her choice was interpreted to mean that when she did not have any knowledge and experience about what to do, she could not employ reflection in her design judgment and tended to follow guidelines.
Britney’s design judgment exercise and development
Britney designed an instruction for Athletic Training students about spine boarding, a procedure that she was already familiar with. Table 6 summarizes Britney’s design judgments throughout her design process and how her design judgment developed over time.
Britney engaged in reflection when she made design judgments, but her reflection fluctuated. Her reflective moments increased as she progressed with her design. In the early stages of her design process she made judgments without deep consideration, but as she progressed and started to develop instruction, she considered more carefully the consequences and implications of her judgments and employed reflection. However, it was determined that when she was making judgments during her formative evaluation, she could not employ reflection because she did not have any prior experience to base the reflection on. Therefore, she tended to follow what she was told.
Annette’s design judgment exercise and development
Annette also designed an instruction on a topic she was familiar with. Her instruction was for Chinese ESL learners regarding how to pronounce certain sounds. Table 7 summarizes her design judgments throughout her design process and how her design judgment developed over time.
Annette’s judgments made with reflection varied; when she had an idea about what would work based on her experience, she did not employ reflection at all but relied on her prior experience. However, in some instances she employed reflection later on. For instance, while she quickly made the judgment about designing instruction on pronunciation without considering the consequences, later she considered who the learners would be and what would be instructed and then made judgments with reflection. It seems she made her initial judgments without considering all of the consequences of this approach, but immediately afterward she reflected on them and experimented with what she might do and what the consequences of those strategies would be. Unlike others, because she had some prior experience with evaluation, she created a method of her own when conducting her formative evaluation in addition to following the guidelines provided in class. Then, she made judgments on using different ways of collecting data in addition to the suggested ones by reflecting on what would help her evaluate the effectiveness of the instruction. This demonstrated that if ID students have prior knowledge and experience to base their reflection on, they may make their judgments with reflection.
Aaron’s design judgment exercise and development
Aaron designed instruction for journalism students covering basic design elements. Like others, he selected a topic and learner group he was familiar with. Table 8 summarizes his design judgments throughout his design process and how his design judgment developed over time.
Aaron made most of his design judgments with reflection when he had freedom of choice. In the early stage of his design process when he framed the design, he quickly made his judgment without considering its consequences. Later, however, when he decided on the content, he employed reflection when he made his judgments. With the knowledge gained from his teaching experience, he shaped the design with his judgment, which was made with reflection. At the end of the process, when planning the formative evaluation, he followed the course guidelines due to his lack of experience and did not make judgments with reflection. Thus, the choices that he made demonstrate that he used reflection in making his judgments when he had prior experience and knew what kinds of choices to make; and when he did not have knowledge or experience to inform his decisions, he tended to abide by the course guidelines.
Moreover, it was recognized that he did not make design judgments with reflection when he needed to meet specific requirements. In the cases of framing both the needs assessment and the assessment, he abided by the specific requirements of the course and also followed suggestions in the form of examples that were presented in the course material. For instance, in framing the needs assessment, he was given examples of certain technologies for collecting data and certain ways of presenting the findings. In that case, his judgment was to follow the guidelines, and he did not employ reflection. However, when developing the instrument and collecting the data, he appeared to engage in reflection along with his judgment. Similarly, when he decided on the assessment items, he was encouraged to use certain types of questions. In that case, he framed his design based on the given requirements and could not incorporate reflection into his judgment. However, when developing the items, he appeared to make a judgment with reflection when he was considering what questions to use and what the appropriate way to present those questions would be. This is an indication that providing too much guidance might have hindered him in making design judgments with reflection.
Jennifer’s design judgment exercise and development
Jennifer designed instruction for high school students to learn music terms. Her choice of topic and learner group was convenient as well, reflecting audience and content she was familiar with. Table 9 summarizes Jennifer’s design judgments throughout her design process and how her design judgment developed over time.
Jennifer made her design judgments with reflection for the most part. Even at the beginning of the process when she framed the design and decided on the focus of the instruction, she engaged in reflection and considered the consequences of her choices. Later, she continued making judgments with reflection and took responsibility for her design. However, when she planned the formative evaluation, she seemed to reduce the use of reflection when making judgments and tended to follow the given guidelines. This indicates that in the early stages, she reflected with the help of her prior knowledge and experience in teaching and learning but because she did not have much experience in formative evaluation, she elected to follow the course guidelines. However, it was identified that when she was working on the details of the formative evaluation, once she had decided what she needed to do, she started engaging in reflection again.
Megan’s design judgment exercise and development
Megan designed instruction for undergraduate students on how to systematically solve a problem, a process that she was familiar with. Table 10 summarizes Megan’s design judgments throughout her design process and how her design judgment developed over time.
Megan hurried through her judgments at first but engaged in reflection later. When she conducted her project she usually engaged in reflection when she made her judgments even though she did not incorporate reflection into all of her judgments from the beginning. She was open to learning new things, and it seemed that she employed reflection when she decided the content of the instruction. In other cases, she seemed not to think through the evaluation, her judgment was to follow the guidelines. Even though she performed the program evaluation at the end of the course, she still tended to follow guidelines and did not employ reflection.
Husain’s design judgment exercise and development
In the course Husain was in, he was required to complete two design projects. He selected the topics he was familiar with. For the first project, he designed instruction on business emails for Chinese learners. For the second project, he and his team started with a different idea but then designed instruction on how to create a rubric. Table 11 summarizes Husain’s design judgments throughout his design process and how his design judgment developed over time.
Husain made judgments relying heavily on research, his own experience, course requirements, best practices, and suggestions. His judgments involved reflection in some cases but not all the time. He was the only student who had some prior experience in the field, and his prior experience might have caused him to make judgments quickly without much consideration because he might have reflected previously on judgments and found considering them again to be unnecessary. Therefore, that might have made it difficult to capture whether his judgments were made with reflection. Because his initial level of experience in this course was different from that of others, his pattern of making judgments with reflection diverged from that of the others. In some cases, he felt he had to restrict his judgment to meet the course requirements. In other cases, he might have intentionally put his judgment aside because he was working in a group and did not want to impose his thoughts on other team members. Further, he might have constrained his judgment because he knew this was a course project and a minimum amount of effort was enough to showcase his ability to design instruction.
Garry’s design judgment exercise and development
Garry was also required to complete two design projects in a semester. In his first project, he designed instruction for second and third grade students on temperature. In the second project, he and his teammate designed instruction on how to 3D print. His second project was on a topic he did not know anything about. He first learned about the topic; then designed an instruction on it. Table 12 summarizes Garry’s design judgments throughout his design process and how his design judgment developed over time.
Garry made most of his judgments with reflection. Especially, in his second design project, he considered the consequences of his actions and thought through the implications of the design. When he framed the design, he considered that instructing learners on how to 3D-print would be too general, and that the learners might create individual designs. To eliminate issues related to designing and focus on the process of printing, he considered narrowing the focus down to printing a nametag. Even at the beginning of his design process, he thought through what possible issues might arise during implementation and how to overcome them before they occurred. Because he was working with a teammate, some of his judgments were made in cooperation with his teammate. His prior knowledge and experience helped him employ reflection when he made his judgments. For instance, when structuring the instruction, he considered reducing the cognitive load during the course of the instruction. He thought through how he could help learners to process their new knowledge easily. As opposed to others, when conducting the evaluation, he did not strictly follow the guidelines but took a different approach. All of this highlights the fact that in his design project, most of the time he employed his reflection when he made judgments.
Also, it was noticed that his design process was different from that of others. First, he decided to design an instruction on a topic he was interested in but had no knowledge of. Thus, he made an effort to learn the procedure about which he would design his instruction. His experience in learning the instructional procedure influenced his judgment on structuring the instruction. Because his project was the only one that would be implemented in the real world, this might have encouraged him to conduct pilot tests and revise the instruction.
Discussion and recommendations
These ID students’ design judgment exercise and development revealed themes regarding (1) the nature of exercising design judgment, (2) influences on exercising and developing design judgment, and (3) the nature of design judgment development. These will be discussed together with the existing literature, and based on the findings, recommendations will be made for ID education to support students’ design judgment development.
Nature of exercising design judgment
ID students make design judgments constantly throughout their ID process regardless of whether they make them with or without reflection. These students’ design process also revealed that they make different kinds of design judgments concurrently, and that some of the judgments overlapped. These findings were consistent with the theory regarding use of design judgment (Nelson & Stolterman, 2003; 2012) as well as empirical studies (Demiral-Uzan, 2015; Farmer & Koehler, 2022; Gray et al., 2015; Parsons et al., 2020).
ID students in this study were identified making clustered and layered design judgments as consistent with prior research of designers in practice (Boling et al., 2017a, 2017b; Gray et al., 2015). The excerpt shown in Table 4 demonstrates that this student is making multiple kinds of design judgments at a certain moment while designing. There may be times in design where designers do make a single kind of design judgment, but other times where they make multiple kinds at the same time. Farmer & Koehler (2022) indicated that even though they reported their design judgments separately, their experience was complex and design judgments were interdependent as other researchers have reported (Gray et al., 2015; Parsons et al., 2020) and as found in this study. Similarly, during design processes, some kinds of design judgment are foregrounded, while others may happen at the background. In Table 4, when the student makes a navigational judgment in the foreground when deciding to continue searching for resources, she makes other kinds of judgments in the background when evaluating the material she already has, considering the quality and relevance of the materials for learners. This demonstrates how design judgments are layered. While the focus of this study was not to identify how clustered and layered these ID students’ design judgments were, it was interesting to note and to observe that students’ exercise of judgments happens similarly to observations of practicing designers. It may be that certain clusters of judgments naturally form during design, but further research is needed to know what making layered and clustered design judgments may say about either design judgment or about students’ design judgment development.
Holt (1997) has indicated that being aware of making design judgments is a pre-condition for developing design judgment. However, design judgment is not explicitly discussed in ID education (Korkmaz, 2011) and in this study we also identified that these students were not encouraged explicitly to use their design judgment and nor were they explicitly aware of using their design judgment even though they were clearly doing so. It is worth considering how their exercise of design judgment might be different if they were explicitly encouraged or guided to recognize, name and embrace their design judgment. We may be able to increase awareness by using common language and implementing frame experiments in ID courses to teach design judgment and to help students explicitly identify the judgments they make in design (Boling et al., 2022).
Influences on the exercise and development of design judgment
Experience, as well as external factors such as course requirements and practice, influence the exercise and development of design judgment. We see from these ID students’ design process that they used their experiences in their design judgments and reflections, and that these experiences shaped their design judgments. This finding is consistent with recent research about ID practitioners’ use of prior experience in design (Stefaniak & Xu, 2020; Stefaniak et al., 2022), and in situations where they do not know with certainty what to do (Stefaniak et al., 2018). However, we also see in this study that students have a limited repertoire of experience to rely on for their judgments. In three out of four ID courses, students were working on one design project for an entire semester. The experiences they were relying on were either their prior teaching experiences and/or their learning experiences. They were not referring to their design experiences. Design theory states that design judgment is dependent on experience (Nelson & Stolterman, 2012) and that design judgment is reflection together with experience (Schön, 1983). Pauwels et al. (2013) have highlighted that designers’ experience are gained through in designing and already accumulated ones. Likewise, Korkmaz (2011) found that instructional designers make design judgments by using their experiences as these ID students did, especially in their reflections. The critical role of experiences raises the question of how we address students’ existing experiences and how we help them to accumulate as much experience as possible to support their design judgment exercise and development. Although researchers suggest using design cases (Boling, 2010) to help students to accumulate as much experience as possible, students in this study were not given any design cases in their ID courses. We note that many students are exposed to teaching cases (e.g., Ertmer et al., 2019), which are fictionalized narratives crafted to raise specific questions about design projects; these are different than design cases, which are authentic descriptions of designs produced for the purpose of increasing the store of experiences for many designers (Howard et al., 2012). Some students in this study did have an opportunity for peer review in class which helps them to see the parts of a design that they had not considered before, encouraging them to reconsider previously made design judgments and to make further judgments. Peer review also provided opportunities for students to expand their repertoire of design situations by reviewing the designs of others, as observed by Korkmaz (2011) who indicated that instructors thought peer reviews help students to develop design judgment. In ID education practice, we may emphasize and expand teaching strategies such as integrating design cases, encouraging development of design cases to build value for this form of developing judgment, and peer review to further support students’ deliberate and explicit development of design judgment. However, we do need further investigation on how an expanded repertoire of design experiences may make a difference in the development of design judgment, particularly accelerating it in a field where most students still begin their design studies at the graduate level with a shorter time for this development than in fields where design instruction begins at the undergraduate level.
It is evident in this study that the supplied design project requirements influenced how these ID students exercised design judgment. As Gray et al. (2015) noted, instructional designers’ judgments were shaped by external factors including the design environment, the role or position of the designer, the project, the client and other external constraints; likewise, these students’ design judgments were shaped by the project requirements determined by the instructors of the courses. Pre-determined requirements such as the method of delivery, the duration of the instruction, the type of learning and guidelines for certain design activities shaped these students’ design judgments in designing. In some cases, they seemed to adjust their design judgments to meet these requirements. In other cases, these requirements prevented them from making design judgments and discouraged them from reflecting on the design process. These pre-determined requirements and supplied guidelines might have limited their development in making design judgments. For instance, the course Aaron was enrolled in employed an ID model approach in which project deliverables were aligned with a specific model starting with needs assessment and ending with a formative evaluation plan. Students in this course were required to design a self-paced online instructional module that would not exceed 15 min in length. They were encouraged to design their instructional modules as interactive presentations that learners could navigate by themselves. In addition, students were given guidelines in the course. Aaron stated in the interview “we were asked to create assessment items that were either true/false or multiple choice, and so it was a little bit limited in creating objectives. And I say that was difficult because I think with my particular content there are so many different ways that you can assess creating a design”. This given guideline seemed to limit Aaron’s design judgment. On the other hand, Megan was designing in a course using a holistic design approach in which students were expected to design an instructional program with participant and facilitator materials, assessment items, and other unspecified resources. She was required to write preliminary, detailed design reports and a program evaluation report. In this course, students were not given any requirements in terms of delivery method or length of the instructional program. Her interview revealed that she received support as needed rather than receiving detailed requirements in advance. In relation to creating assessment items, Megan stated that she did not want to create a multiple choice test for assessment but wanted to use “a scenario-based kind of practice and assessment”. She thought through what kind of scenario-based assessment she could use. She shared that she first thought about giving students a problem to solve and observe them when they were doing but then she thought this would not be feasible and changed her perspective deciding to describe a problem that someone else was having, asking students to help this person solve that problem. She thought that with this perspective she would be able to assess what aspects students attended to when helping solve the problem. She was able to carry out this idea in her design. However, in Aaron’s case, he was not given choice to explore new ideas and he felt that he needed to comply with what he had been asked to do. These two cases illustrate how starting with detailed guidelines and suggestions on what to do may prevent ID students from using design judgment to explore and select options. As educators, we may think that novices will need guidelines to be able to know what they are doing. Clearly, taking guidelines away from instruction design education may cause students to feel insecure and lost in the process. Guidelines may have a role in providing support for those who do not know what to do, but due to detailed guidelines, students may be trapped in the novice level and it may be hard for them to break through (Boling, 2016). Therefore, it may be worth considering both the timing and the nature of guidelines. Should the guidelines tell students what to do before they start working, or be given as needed to support them in the process? Researchers have suggested that a coaching or apprenticeship model in which students approach the full complexity of design immediately to support students’ development (Ertmer & Cennamo, 1995; Schön, 1987; Stefaniak, 2017; Tripp, 1994). Ertmer et al. (2009) recommended pairing novice ID students with experts to support problem solving skills; similarly, experienced ID students may be paired with novice ID students to support their design judgment development or, as in studio formats, novice and advanced students may study in the same course. Through coaching, novice students may have the opportunity to explore, reflect, think more deeply than they would otherwise and make design judgments as Stefaniak (2017) has suggested. Working in a coaching or studio model may also help novice designers break through unseen barriers that may be implied by the requirements and guidelines intended to scaffold them in design projects (Boling et al., 2017b). However, careful attention is needed because if instructors or the more experienced students do not know much about design judgment and do not encourage making design judgments but instead simply offer tips and provide shortcuts, coaching may not help in supporting design judgment development.
Nevertheless, as the students in this study progressed in design, they most often started with non-reflective judgments but were demonstrably able later to make design judgments with reflection and to consider interconnections in design. As Smith (2008) has suggested, novice designers need to exercise judgment to be able to learn how to make better judgments. These students’ design judgments were clearly developing with designing and practicing design, and it is evident in this study that working on design projects did encourage ID students to make design judgments and reflect, even without explicit reference to design judgment in their classes. However, most students in this study still had limited opportunities to practice their design judgment except while working on a single design project across the semester. It is possible that if they had more opportunities, their design judgment could have developed more. As seen with Garry who had two project opportunities for design practice, he was able to make more reflections during exercise of his design judgment even at the beginning of the design process. It is important to remember that students’ design judgment development will not be accomplished with a single course in a semester, whether they work on one project or two. Instead, students need to be given opportunities to exercise and develop their design judgment with repeated practice throughout their professional education. Repeated practice is, of course, a staple instructional strategy for both simple and complex performances (Pellegrino, 2004), but in two-year instructional design programs there may need to be curriculum adjustments to provide repeated opportunities for students to grapple with various design situations and, as Stefaniak (2015) has suggested, to provide these as early as possible in their programs.
While Lawson and Dorst (2009) have stated that for development of design expertise students should be assigned a series of design projects with complexity increased over the series, consistent with Reigeluth’s (2013) elaboration theory and simplifying conditions, Nelson & Stolterman (2012) and others (Boling, 2016; Orr & Shreeve, 2017; Stefaniak, 2017) have suggested that students should be given design tasks that are open-ended and demand design judgment. They highlight the need for ID and design students to deal with difficult design scenarios and practice designing instruction in varied contexts. It will be worth asking in future studies: do multiple design projects and fully complex projects function in ID education as pedagogical theory suggests they should?; how should such projects be framed to support students’ design judgment?; how many projects might be enough for students to develop their design judgment to a desirable level in a master’s program?; which contributes most to students’ design judgment development—the number of projects they engage in or the nature of the design projects or are both equally important?
Multiple means have been suggested to provide opportunities for students to practice and support instructional design in general. Larson & Lockee (2009) have suggested that instructors can let students experience the context in which they will work later; this is consistent with a view of design education as a bridge to professional practice (Brandt et al., 2013). Case-based learning is another strategy which has been used to improve the preparation of ID students (Bennett, 2010). Bannan-Ritland (2001) has described applying an action learning approach to teaching ID, explaining that each ID situation brings unique challenges for designers in dealing with ambiguity and the constraints of that design situation. In all of these approaches we suggest that adding explicit focus on design judgment with reflection might accelerate and/or deepen development of students’ judgment which is likely already occurring.
In addition to these teaching strategies, researchers have suggested incorporating a designer mindset (Stefaniak & Reese, 2022), using frame experiments as an instructional method (Boling et al., 2022) and fostering design thinking and design judgment (Tracey & Boling, 2014) in ID education. Studio-based education (Boling et al., 2017a, b) may provide opportunities for all the recommendations resulting from this study and may be worth considering for helping students to recognize their own design judgment, designer perspective and paths to develop their design judgment.
Nature of developing design judgment
Development of design judgment was personal and perceptible for these ID students over the course of the semester. Consistent with design judgment and development being theorized to be unique to each designer (Holt, 1997; Nelson & Stolterman, 2012), design judgment and its development in these ID students did manifest in unique ways, individual to each participant in the study. This was not only because they were in different design situations and thus confronted a variety of design problems; it appeared also to be because they had different approaches toward designing and different knowledge sets and experiences coming into class. Lawson (2006) has indicated that, indeed, designers have different approaches and views about how to design and they bring those into design. Even the students who had a similar number of years of teaching experience and who were designing instruction on a topic they had taught before made design judgments individual to themselves.
Korkmaz (2011) has suggested that working on design projects may foster ID students to develop design judgment because instructors in her study reported that they expected students to exercise design judgment during design projects. However, Dorst & Reymen (2004) argued that a ‘learning by doing’ model is not enough to develop the abilities needed for design. Lawson & Dorst (2009) also argued that simply doing projects does not guarantee learning and development, that skills and values are needed as well. This study suggests that even though class projects may not be enough to develop all required design abilities, working on design projects does provide the opportunity for students to exercise design judgment and develop design judgment to some degree. The majority of the students’ design processes in this study were aligned with process steps prescribed to them. Structuring their activities by process steps may have discouraged them from going outside the provided guidelines and design model, preventing them from exploring new approaches to design, increasing their comfort level but reducing the chances for them to exercise, and therefore develop their judgment. Also, most students selected a topic that they were familiar with for which to design instruction. While they often followed unreflective judgments with various forms of reflection, coming to the design situation with a frame already in mind may have prevented them from making design judgments with sufficient reflection. Their design judgment development might, consequently, have remained limited.
Lawson & Dorst (2009) have argued that designers’ development starts with the novice following strict rules when dealing with design problems. These ID students’ design process reveals that most of them still tended to follow the rules by the end of the course. In that sense, those students remained close to the novice level of performance and preferred to stay in a safe zone when they were provided with requirements and guidelines on what to do and how to do it. They tended to follow these and perform according to supplied processes, especially when they did not have prior experience to rely on. As Nelson & Stolterman (2012) have discussed, guidelines might serve as the guarantor-of-design that students may search for. On the other hand, a few of the students did stretch the rules based on their unique design situations. Being sensitive to design situations and operating on exceptions to the rules is described as the performance of advanced beginners (Lawson & Dorst, 2009). In that sense, some students did seem to move beyond the novice level.
In this study, most students’ design judgment with reflection fluctuated. Their pattern of making design judgments did reveal that their ability to make design judgments with reflection changed. For instance, while they made design judgments without consideration at the beginning of the design process, as they progressed through their projects, they returned to the judgments they had made earlier, reflected on those, and operated on the basis of some interconnected considerations. This shows that as they progressed in design, their judgments evolved and were made with reflection and consideration and signaled that their ability to make design judgments had developed over the semester. On the other hand, a few students seemed to make design judgments with reflection throughout the design process; it was difficult to determine in these cases whether their design judgment with reflection had changed over the semester or whether it was a tendency toward reflection on their part regardless of making design judgments. Regardless, providing opportunities for students to reflect deliberately could help them to recognize their own design judgment at work and to develop it deliberately. As an example, Tracey et al. (2014) found reflective writing assignments beneficial for novice designers to make meaningful reflections on their designs. Another strategy might be to use just-in-time prompts given when students are engaged in designing to help students reflect in situ and work to gain the habit of reflective thinking while making design judgments (e.g., frame experiments as suggested by Boling et al., 2022). Future study of strategies such as these could illuminate productive pathways to moving novice designers further in their development during instructional design courses.
Limitations and future research
This study explored eight ID students’ design judgment exercise and development. The number of participants in this study was limited; however, diversity among the cases included ID courses with different structures at various institutions. With the limited number of participants, we were, however, able to give good attention to the rich data collected. The aim of the study is, of course, not to generalize to all ID students or even to other students in the courses participants came from.
Another limitation of the study involves not observing students while they were actively designing owing to the distance between the different institutions. Additionally, when the interviews were conducted students had already made some of their design judgments and articulating their thought processes from the time when those judgments were made was sometimes challenging for them. To further explore students’ exercise and development of design judgment, the next step would be conducting studies that observe students directly in class, in design meetings and in other design activities using the lens of design judgment. Direct observation of students’ design activities would enable researchers to examine their design character, perspectives, and judgment in the moment together with changes in their ability to make those judgments.
It is evident in this study that working on design projects allowed for students to exercise design judgment; however, the design projects they were engaged in might have influenced their design judgment development. Further research is needed to investigate the role of multiple design projects in the development of design judgment and how design projects need to be framed to support design judgment development. We found that external factors such as course requirements, given guidelines and suggestions might have prevented students from making some design judgments and therefore feel that further research is needed to explore the role of course requirements and guidelines on students’ development of design judgment and reflection. Moreover, methods for making explicit what design judgment is and how to reflect upon it should be investigated. This study explored ID students’ exercise and development of design judgment for one semester. Longitudinal studies also need to be conducted to capture students’ development over longer periods of time, especially when they are asked to practice complex design early and repeatedly throughout their educational experience.
Data availability
Not applicable.
References
Abbott, D. (2020). Intentional learning design for educational games: A workflow supporting novices and experts. In: M. Schmidt, A. A. Tawfik, I. Jahnke, & Y. Earnshaw (Eds.). Learner and user experience research: An introduction for the field of learning design & technology. EdTech Books. https://edtechbooks.org/ux/11_intentional_learn.
Bannan-Ritland, B. (2001). Teaching instructional design: An action learning approach. Performance Improvement Quarterly, 14(2), 37–52.
Batchelder, E. A. (1914). Design in theory and practice. Macmillan.
Bennett, S. (2010). Investigating strategies for using related cases to support design problem solving. Educational Technology Research and Development, 58(4), 459–480.
Boling, E. (2010). The need for design cases: Disseminating design knowledge. International Journal of Designs for Learning, 1(1), 1–8.
Boling, E. (2016). Teaching the complex performance of instructional design: Why we cannot use the (existing) tools of instructional design. In A. Carr-Chellman & G. Rowland (Eds.), Issues in technology, learning, and instructional design: Classic and contemporary dialogues (pp. 81–86). Routledge.
Boling, E., Alangari, H., Hajdu, I., Guo, M., Gyabak, K., Khlaif, Z., Kizilboga, R., Tomita, K., Alsaif, M., Bae, H., Ergulec, F., Lachheb, A., Zhu, M., Basdogan, M., Buggs, C., Sari, A., & Techawitthaychinda, R. (2017a). Core judgments of instructional designers in practice. Performance Improvement Quarterly, 30(3), 199–219.
Boling, E., Gray, C. M., & Lachheb, A. (2022). Inscribing a designer mindset to instructional design students. The instructional design trainer’s guide (pp. 18–28). Routledge.
Boling, E., Schwier, R. A., Gray, C. M., Smith, K. M., & Campbell, K. (Eds.). (2017b). Studio teaching in higher education: Selected design cases. Routledge.
Brandt, C. B., Cennamo, K., Douglas, S., Vernon, M., McGrath, M., & Reimer, Y. (2013). A theoretical framework for the studio as a learning environment. International Journal of Technology and Design Education, 23, 329–348.
Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101.
Chang, Y. K., & Kuwata, J. (2020). Learning experience design: Challenges for novice designers. In: M. Schmidt, A. A. Tawfik, I. Jahnke, & Y. Earnshaw (Eds.), Learner and user experience research: An introduction for the field of learning design & technology. EdTech Books. https://edtechbooks.org/ux/LXD_challenges.
Christensen, T. K., & Osguthorpe, R. T. (2004). How do instructional-design practitioners make instructional-strategy decisions? Performance Improvement Quarterly, 17(3), 45–65.
Creswell, J. W., & Miller, D. L. (2000). Determining validity in qualitative inquiry. Theory into Practice, 39(3), 124–130.
Cross, N. (2001). Designerly ways of knowing: Design discipline vs. design science. Design Issues, 77(3), 49–55.
Demiral-Uzan, M. (2016). The evolution of view toward design in the field. Concurrent Session at AECT International Convention 2016, Las Vegas, NV.
Demiral-Uzan, M. (2015). Instructional design students’ design judgment in action. Performance Improvement Quarterly, 28(3), 7–23.
Dorst, K., & Reymen, I. (2004). Levels of expertise in design education. In DS 33: Proceedings of E&PDE 2004, the 7th international conference on engineering and product design education, Delft, the Netherlands, 02-03 Sep 2004.
Dorst, K. (2003). Understanding design, 150 reflections on being a designer. BIS Publishers.
Ertmer, P. A., & Cennamo, K. S. (1995). Teaching instructional design: An apprenticeship model. Performance Improvement Quarterly, 8(4), 43–58.
Ertmer, P. A., Quinn, J. A., & Glazewski, K. D. (Eds.). (2019). The ID casebook: Case studies in instructional design. Routledge.
Ertmer, P. A., Stepich, D. A., Flanagan, S., Kocaman-Karoglu, A., Reiner, C., Reyes, L., et al. (2009). Impact of guidance on the problem-solving efforts of instructional design novices. Performance Improvement Quarterly, 21(4), 117–132.
Ertmer, P. A., Stepich, D. A., York, C. S., Stickman, A., Wu, X. L., Zurek, S., & Goktas, Y. (2008). How instructional design experts use knowledge and experience to solve ill-structured problems. Performance Improvement Quarterly, 21(1), 17–42.
Farmer, T., & Koehler, A. (2022). Design judgments in the creation of elearning modules. Journal of Formative Design in Learning, 1, 1–12.
Fortney, K. S., & Yamagata-Lynch, L. C. (2013). How instructional designers solve workplace problems. Performance Improvement Quarterly, 25(4), 91–109.
Gerring, J. (2007). Case study research: Principles and practices. Cambridge University Press.
Gibbons, A. S., Boling, E., & Smith, K. M. (2014). Instructional design models. In J. M. Spector, M. D. Merrill, J. Elen, & M. J. Bishop (Eds.), Handbook of research on educational communications and technology (4th ed., pp. 607–615). New York: Springer.
Gray, C. M., & Boling, E. (2018). Designers’ articulation and activation of instrumental judgments in cross-cultural user research. Special Issue, Co-Design: International Journal of CoCreation in Design and the Arts, 14(2), 79–97.
Gray, C. M., Dagli, C., Demiral-Uzan, M., Ergulec, F., Tan, V., Altuwaijri, A., Gyabak, K., Hilligoss, M., Kizilboga, R., Tomita, K., & Boling, E. (2015). Judgment and instructional design: How ID practitioners work in practice. Performance Improvement Quarterly., 28(3), 25–49.
Hancock, D. R., & Algozzine, R. (2006). Doing case study research: A practical guide for beginning researchers. Teachers College Press.
Holt, J. E. (1997). The designer’s judgement. Design Studies, 18(1), 113–123.
Howard, C. D., Boling, E., Rowland, G., & Smith, K. M. (2012). Instructional design cases and why we need them. Educational Technology, 52(3), 34–38.
Jahnke, I., Schmidt, M., Pham, M., & Singh, K. (2020). Sociotechnical-pedagogical usability for designing and evaluating learner experience in technology-enhanced environments. In M. Schmidt, A. A. Tawfik, I. Jahnke, & Y. Earnshaw (Eds.), Learner and user experience research: An introduction for the field of learning design & technology. EdTech Books https://edtechbooks.org/ux/sociotechnical_pedagogical_usability
Jonassen, D. H. (2008). Instructional design as design problem solving: An iterative process. Educational Technology, 48, 21–26.
Kenny, R. F., Zhang, Z., Schwier, R. A., & Campbell, K. (2004). A Review of what instructional designers do: Questions answered and questions not asked. Canadian Journal of Learning and Technology, 31(1), 9–16.
Kerr, S. T. (1983). Inside the black box: Making design decisions for instruction. British Journal of Educational Technology, 14(1), 45–58.
Kirschner, P., Carr, C., Merriënboer, J., & Sloep, P. (2002). How expert designers design. Performance Improvement Quarterly, 15(4), 86–104.
Korkmaz, N. (2011). How is development of design judgment addressed in instructional design education?. (Unpublished doctoral dissertation). Indiana University, Bloomington, Indiana.
Korkmaz, C. (2018). “ID 2 LXD” from instructional design to learning experience design: The rise of design thinking. In A.-P. Correia (Ed.), Driving educational change: innovations in action. https://ohiostate.pressbooks.pub/drivechange/chapter/id-2-lxd-from-instructional-design-to-learning-experience-design-the-rise-of-design-thinking/.
Lachheb, A., & Boling, E. (2018). Design tools in practice: Instructional designers report which tools they use and why. Journal of Computing in Higher Education, 30(1), 34–54.
Lachheb, A., & Boling, E. (2020). The role of design judgment and reflection in instructional design. In J. K. McDonald & R. E. West (Eds.), Design for learning: Principles, processes, and praxis. EdTech Books.
Larson, M. B., & Lockee, B. B. (2009). Preparing instructional designers for different career environments: A case study. Educational Technology Research and Development, 57(1), 1–24.
Lawson, B. (2006). How designers think: The design process demystified. Architectural Press.
Lawson, B., & Dorst, K. (2009). Design expertise. Elsevier Ltd.
Lincoln, Y. S., & Guba, E. G. (2003). Paradigmatic controversies, contradictions, and emerging confluences. In N. K. Denzin & Y. S. Lincoln (Eds.), The landscape of qualitative research (2nd ed., pp. 253–291). Sage.
Merriam, S. B. (2009). Qualitative research: A guide to design and implementation. Jossey-Bass.
Miles, M. B., & Huberman, M. A. (1994). Qualitative data analysis. An expanded sourcebook (2nd ed.). Sage.
Miles, M. B., Huberman, M. A., & Saldaña, J. (2014). Qualitative data analysis. A methods sourcebook (3rd ed.). Sage Publications.
Murphy, D. (1992). Is instructional design truly a design activity? Education and Training Technology International, 29(4), 279–282.
Nelson, H. G., & Stolterman, E. (2003). Design judgement: Decision-making in the ‘Real’ world. The Design Journal, 6(1), 23–31.
Nelson, H. G., & Stolterman, E. (2012). The design way: Intentional change in an unpredictable world (2nd ed.). MIT Press.
NVivo qualitative data analysis Software; QSR International Pty Ltd. Version 11, 2015.
Onwuegbuzie, A. J., & Leech, N. L. (2007). A call for qualitative power analyses. Quality & Quantity, 41(1), 105–121.
Orr, S., & Shreeve, A. (2017). Art and design pedagogy in higher education: Knowledge, values and ambiguity in the creative curriculum. Routledge.
Parsons, P., Gray, C. M., Baigelenov, A., & Carr, I. (2020). Design judgment in data visualization practice. In 2020 IEEE Visualization Conference (VIS) (pp. 176–180). IEEE.
Patton, M. Q. (2015). Qualitative research & evaluation methods: Integrating theory and practice (4th ed.). Sage.
Pauwels, P., De Meyer, R., & Van Campenhout, J. (2013). Design thinking support: Information systems versus reasoning. Design Issues, 29(2), 42–59.
Pellegrino, J. W. (2004). Complex learning environments: Connecting learning theory, instructional design, and technology. Curriculum, plans, and processes in instructional design (pp. 25–48). Routledge.
Perez, R. S., & Emery, C. D. (1995). Designer thinking: How novices and experts think about instructional design. Performance Improvement Quarterly, 8(3), 80–95.
Polanyi, M. (1966). The tacit dimension. Anchor Books.
Reigeluth, C. M. (2013). The elaboration theory: Guidance for scope and sequence decisions. Instructional-design theories and models (pp. 425–453). Routledge.
Reigeluth, C. M., Beatty, B. J., & Myers, R. D. (2016). Instructional-design theories and models Volume IV The learner-centered paradigm of education. Routledge.
Rowland, G. (1992). What do instructional designers actually do? An initial investigation of expert practice. Performance Improvement Quarterly, 5(2), 65–86.
Rowland, G. (1993). Designing and instructional design. Educational Technology Research and Development, 41(1), 79–91.
Schatz, S. (2003). A matter of design: A proposal to encourage the evolution of design in instructional design. Performance Improvement Quarterly, 16(4), 59–73.
Schmidt, M., & Huang, R. (2022). Defining learning experience design: Voices from the field of learning design & technology. TechTrends, 66(2), 141–158.
Schön, D. A. (1983). The reflective practitioner: How professionals think in action. Basic Books.
Schön, D. A. (1987). Educating the reflective practitioner: Toward a new design for teaching and learning in the professions. Jossey-Bass.
Simon, H. A. (1988). The Science of Design: Creating the Artificial. Design Issues, 4(1), 67–82.
Smith, K. M. (2008). Meanings of "design" in instructional technology: A conceptual analysis based on the field’s foundational literature. (Unpublished doctoral dissertation). Indiana University, Bloomington, Indiana.
Smith, K. M., & Boling, E. (2009). What do we make of design? Design as a concept in educational technology. Educational Technology, 49(4), 3–17.
Stefaniak, J., & Sentz, J. (2020). The role of needs assessment to validate contextual factors related to user experience design practices. In M. Schmidt, A. A. Tawfik, I. Jahnke, & Y. Earnshaw (Eds.), Learner and user experience research: An introduction for the field of learning design & technology. EdTech Books. https://edtechbooks.org/ux/role_of_needs_assessment.
Stefaniak, J. E. (2015). The implementation of service-learning in graduate instructional design coursework. Journal of Computing in Higher Education, 27, 2–9.
Stefaniak, J. E. (2017). The role of coaching within the context of instructional design. TechTrends, 61, 26–31.
Stefaniak, J., Baaki, J., Hoard, B., & Stapleton, L. (2018). The influence of perceived constraints during needs assessment on design conjecture. Journal of Computing in Higher Education, 30, 55–71.
Stefaniak, J., Baaki, J., & Stapleton, L. (2022). An exploration of conjecture strategies used by instructional design students to support design decision-making. Educational Technology Research and Development, 70(2), 585–613.
Stefaniak, J. E., & Hwang, H. (2021). A systematic review of how expertise is cultivated in instructional design coursework. Educational Technology Research and Development, 69(6), 3331–3366.
Stefaniak, J. E., & Reese, R. M. (2022). A Holistic Approach to Teaching Instructional Design. The instructional design trainer’s guide (pp. 1–8). Routledge.
Stefaniak, J., & Xu, M. (2020). Leveraging dynamic decision-making and environmental analysis to support authentic learning experiences in digital environments. Revista De Educación a Distancia (RED). https://doi.org/10.6018/red.412171
Stolterman, E. (2008). The nature of design practice and implications for interaction design research. International Journal of Design, 2(1), 55–65.
Tracey, M. W., & Baaki, J. (2014). Design, designers, and reflection-in-action. In B. Hokanson & A. Gibbons (Eds.), Design in educational technology: Design thinking, design process, and the design studio (pp. 1–13). Springer.
Tracey, M. W., & Boling, E. (2014). Preparing instructional designers: Traditional and emerging perspectives. In J. M. Spector, M. D. Merrill, J. Elen, & M. J. Bishop (Eds.), Handbook of research on educational communications and technology (4th ed., pp. 653–660). Springer.
Tracey, M. W., Hutchinson, A., & Grzebyk, T. Q. (2014). Instructional Designers as Reflective Practitioners: Developing Professional Identity through Reflection. Educational Technology Research and Development, 62(3), 315–334.
Tripp, S. D. (1994). How should instructional designers be educated? Performance Improvement Quarterly, 7(3), 116–126.
Vann, S. W., & Tawfik, A. A. (2020). Flow theory and learning experience design in gamified learning environments. In M. Schmidt, A. A. Tawfik, I. Jahnke, & Y. Earnshaw (Eds.). Learner and user experience research: An introduction for the field of learning design & technology. EdTech Books. https://edtechbooks.org/ux/11_intentional_learn
Vickers, S. G. (1984). Judgment. The ‘Vickers’ Papers (pp. 230–245). Harper & Row.
Wedman, J., & Tessmer, M. (1993). Instructional designers decisions and priorities: A survey of design practice. Performance Improvement Quarterly, 6(2), 43–57.
Williams, D. D., South, J. B., Yanchar, S. C., Wilson, B. G., & Allen, S. (2011). How do instructional designers evaluate? A qualitative study of evaluation in practice. Educational Technology Research and Development, 59(6), 885–907.
Winer, L. R., & Vázquez-Abad, J. (1995). The present and future of ID practice. Performance Improvement Quarterly, 8(3), 55–67.
Yanchar, S. C., & Gabbitas, B. W. (2011). Between eclecticism and orthodoxy in instructional design. Educational Technology Research and Development, 59(3), 383–398.
Yin, R. K. (2014). Case study research: Design and methods (5th ed.). SAGE publications.
Zhu, M., Basdogan, M., & Bonk, C. J. (2020). A case study of the design practices and judgments of novice instructional designers. Contemporary Educational Technology. https://doi.org/10.30935/cedtech/7829
Funding
Open access funding provided by the Scientific and Technological Research Council of Türkiye (TÜBİTAK).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Declarations
Ethical approval
This manuscript is part of a dissertation submitted to the faculty of the University Graduate School in partial fulfillment of the requirements for the degree Doctor of Philosophy in the School of Education, Indiana University.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Demiral-Uzan, M., Boling, E. Instructional design students’ design judgment development. Education Tech Research Dev (2024). https://doi.org/10.1007/s11423-024-10361-1
Accepted:
Published:
DOI: https://doi.org/10.1007/s11423-024-10361-1