Design tools in practice: instructional designers report which tools they use and why

Instructional design (ID) tools—tangible and theoretical/methodological tools—play an important role in instructional design (ID) practice. As Gibbons, Boling and Smith observed in 2014, ID tools, mainly models and principles, aim to cover all design activities. Despite the importance of ID tools implied by this, minimal attention has been paid to the subject in instructional design and technology (IDT) scholarship, at least to the practitioners’ views and experiences with design tools. A few exceptions are found in the IDT literature that shed a light on ID tools from the practitioner’s perspective (e.g., Dicks and Ives 2008; Sugar et al. 2011, 2012; Yanchar et al. 2010).

Research on ID practice has been witnessing what Gray et al. (2015) called a shift gradually taking place. In fact, studies that are adopting a descriptive approach to ID practice started to appear since 1990s. Rowland’s (1992) rigorous descriptive study to investigate what instructional designers do marks an early point in this shift. Similar studies that focused on ID practice and adopted a descriptive approach have been widely cited in IDT literature. Based on a citation network analysis conducted by Cho et al. (2011), Cho and Park found in 2012 that among the top 20 most influential papers published in Performance Improvement Quarterly were four studies on ID practice (Perez and Emery 1995; Rowland 1992; Wedman and Tessmer 1993; Winer and Vázquez-Abad 1995). The value of such studies originates from their common goal: generate a rich description of ID practice that can inform ID education and practice.

An integrative review of 20 publications between 1990 and the present that were identified as discussing instructional design tools reveals two knowledge gaps. First, studies that examined ID tools in practice did not take a comprehensive approach to study ID tools. In fact, studies on ID tools focus either on computer-based tools or theoretical/methodological tools, not both combined. Most notably, Sugar’s (2014) synthesis and critique of literature on ID practices highlights a similar idea—a lack of “holistic perspective of current ID practices from the viewpoint of an ID professional and an ID student” (p. 106).

Secondly, articles describing ID practitioners’ views and experiences (i.e., descriptive) with ID tools were found to be minimal. With the exception of six studies (Dicks and Ives 2008; Roytek 2010; Sugar et al. 2011, 2012; Svihla et al. 2015; Yanchar et al. 2010) that include the practitioners’ voice, other studies reviewed may have focused on tools (i.e., prescriptive) but ignored the views of ID practitioners.

Our perspective is that if prescriptive studies on ID tools (and on ID practice in general) continue to dominate IDT literature, the ID field will not progress and flourish like other fields of design (e.g., architecture, graphic design, engineering design), where a richer understanding has been promoted through descriptive inquiry that captures design in situ (Smith and Boling 2009; Sugar 2014). With descriptive scholarship on ID practice, an understanding of the field and its complex nature should help ID educators consider what ID tools to teach, how to teach them and how to use them. Without such study on this important topic, we can face the risk of seeing tools developed for ID practitioners not used resulting in time and money being wasted and we will “miss opportunities” (Boling et al. 2017, p. 200).

Morrison et al. (2011) define instructional design (ID) as “the process for designing instruction based on sound practices” (p. 6). The practice of a profession, (ID in this case) as argued by Crouch and Pearce (2012) refers to activities that are standardized and approved by an agreed-upon professional authority (e.g., professional association). However, Crouch and Pearce (2012) provide an alternative definition of practice that is the most suitable for the purpose of this study—“practice is not necessarily indicative of how things ought to be” (p. 34). ID practice with this alternative definition is viewed to be intertwined with ideas and theories originating from personal experiences and social norms, not only those prescribed by a given professional authority.

Boling and Gray (2015) have posited a comprehensive definition of ID tools as “methods, tools, techniques, and approaches, as well as design models, design and learning theories, and principles” (p. 112). Under this definition, the term tool is used in its broad sense to refer to actual tools (e.g., computers), methods (e.g., prototyping), techniques (e.g., negotiation), and approaches (e.g., client-oriented). This definition is partially inspired by the works of Stolterman et al. (2009), and it is underpinned by the following two assertions: (1) there are a variety of tools besides ID models; and (2) there is no established order that guides the usage of ID tools. Empirical and conceptual investigations of ID tools reviewed in the literature were found to share the same elements of this comprehensive definition of ID tools, and not offer an alternative definition.

Studies reviewed were identified following Torraco’s (2016) method in conducting an integrative literature review. Three online databases were consulted to search relevant literature (Google Scholar, Academic Search Premier, and ERIC). Key words used were “instructional design tools”, “instructional designer tools”, “designerly tools”, and “instructional designer technology”. Four other publications were added to the review of the literature. These four publications have a significant value in providing foundational knowledge about the research topic (Sugar et al. 2011, 2012; Tripp 1991; Sugar 2014). Two selection criteria were adopted to narrow the search into more suitable results. These criteria were (1) scholarly peer-reviewed journal articles, books and book chapters that discuss ID tools empirically and conceptually, (2) conference proceedings that report empirical and conceptual types of research on ID tools.

Twenty publications that met the above criteria were selected for review; they were read and summarized in detail. Three broad ideas emerged from this review: significance of ID tools, computer-based ID tools, and theoretical/methodological ID tools. These three broad ideas were transformed into three themes, which were later used to report a summary of literature on ID tools.

In his (2014) review of studies on instructional design practices, Sugar strongly recommends an “all-inclusive” approach to such studies, exploring everything that designers do rather than framing studies through a limited number of pre-established and prescriptive process steps or concepts (p. 106). He further stresses the importance of the practitioners’ views and experiences, included in studies available at the time of his review. Several recent empirical studies on ID practice that focus on the practitioner voice are premised on the idea that the practitioner is the guarantor of design (Nelson and Stolterman 2012)—the one responsible for the success and/or the failure of a design, rather than the model or theory or process being used by the designer (Boling and Gray 2015; Gray et al. 2015). This concept characterizes the contemporary philosophy of ID (Carr-Chellman and Rowland 2016) which stands in contrast to the classical view of ID that focuses on models, theory, data and systematic approaches (Carr-Chellman and Rowland 2016; Smith and Boling 2009).

Design tools in HCI (which may be considered a neighboring field of practice) have been studied from the practitioners’ perspectives since the early 2000s. Stolterman et al. (2009) and Stolterman and Pierce (2012), studied the relationship between interaction designers and their tools as they are practiced in situ. They relied on a qualitative, grounded theory approach to data collection and analysis proposing a framework, the Tools-in-Use Model (Stolterman et al. 2009), to study design tools in practice and call on all fields of design to investigate their own practices and the tools in used in practice. They pose serious questions like, “How are tools taught in our design education? [and] Do designers feel as if they are using the ‘wrong’ tools in our field, and if so, why?” (Stolterman et al. 2009, p. 11).

A heavy emphasis on the evaluation of computer-based ID tools is also notable in the IDT literature. Several contributors to this emphasis may be: computer-based ID tools are generally costly to buy and to maintain (Merrill 2001) and therefore warrant attention; there are also many tools of this type available (Chapman 1995); and the many “technology effect” studies emerging after passage of the No Child Left Behind Act in 2001 (Ross and Morrison 2014) which included increased focus on ID tools. The constructs underlying computer-based tools can be seen, as in the four publications in the ETR&D 2002 special issue—to rely on a classical view of ID centered on process and guided by design models and on argumentation rather than empirical study in the field. Gustafson’s (2002) critique of these publications calls for “systematically evaluating the effectiveness and appeal of the education and training that result from using ID (tools)” (p. 65) [emphasis added].

In summary, ID tools have been discussed since the 1970s (Kearsley 1977). Each selected publication for review has indicated the significance of ID tools in ID practice. This significance is exhibited through an emphasis on evaluating computer-based ID tools and studying theoretical/methodological ID tools in ID practice. Despite the rich knowledge currently published on ID tools, two major knowledge gaps are noted; (1) There is a lack of inclusive research on ID tools. (2) There is a minimal focus on the practitioner’s views and experiences regarding ID tools. Both noted knowledge gaps are supported by Sugar’s (2014) comprehensive synthesis and critique of studies on ID practice.

The purpose of this study is to investigate ID tool use from the perspective of ID practitioners. This study adopts a broad definition of ID tools that include methods, tools, techniques, and approaches (Boling and Gray 2015; Stolterman et al. 2009) and uses the lens of designerly tools (Stolterman et al. 2009). This perspective suggests that tools open to flexible use by designers will see more use and enjoy more acceptance than those constraining the design process. Authentic and rich descriptions of ID practice in its own terms (vs those of prescriptive scholarship or theory) can inform educators to better prepare ID practitioners; the focus on tool use is one part of investigating ID practice.

Tracey and Boling (2014) made the case in 2014 that IDT scholarship predominantly aims to measure the work of ID practitioners “against what academics think they should be doing” (p. 658), which has resulted in a limited understanding of how instructional design actually occurs in situ, and which limits preparation of practitioners in the field. Studies have begun to appear in the field which approach ID practice on its own terms, descriptively versus prescriptively, by focusing on the practitioners’ views and experiences (Christensen and Osguthorpe 2004; Cox and Osguthorpe 2003; Gray et al. 2015; Kenny et al. 2005; Rowland 1992; Visscher-Voerman and Gustafson 2004) but little attention has been paid to instructional design tools from the perspective of ID practitioners (e.g., Dicks and Ives 2008; Sugar et al. 2011, 2012, Sugar 2014; Yanchar et al. 2010). When tools are addressed in the IDT literature, emphasis is placed on evaluating their impact on the outcome of ID, assuming that ID is defined as the application of theory grounded and guided by design models. Studies like these are exemplified in a special issue of Educational Technology Research and Development (2002). This view of ID tools narrows our understanding of practice in the field rather than expanding it.

Descriptive statistics techniques were used to analyze survey data and produce simple frequency counts of ID tools selected/entered in the survey. Counts of ID tools selected from the initial listed were tabulated automatically via Qualtrics; ID tools entered in response to the open-ended question were tabulated in Excel through color coding and data sorting.

Merriam and Tisdell (2015) open coding process was used to analyze interview data. Each transcript was read multiple times and segments that were found to be relevant to the study were highlighted. Initial notes were taken directly on the transcripts and provisional themes noted. Themes were highlighted and revisited again while reading the interview transcripts; these themes were further grouped (axial coding) into more general themes with an emphasis on the framework of designerly tools developed by Stolterman et al. (2009). After consultation between the researchers, the transcripts were re-analyzed using the general themes thus derived. Both frequency counts of these themes across the transcripts and holistic appreciation of respondents’ meanings informed the final findings and discussion.

The survey data show that among almost 100 ID practitioners a large number and variety of ID tools are used, many of them computer-based. The emphasis on computer-based tools corresponds with Roytek’s (2010) study that highlights wide use of computer-based technologies among instructional designers. Roytek (2010) also identifies an important ID competency that requires the knowledge of authoring tools (i.e., computer-based software). Current modalities of learning, including online and blended learning, may contribute to this frequent use. Conceptual and methodological tools are reported frequently as well, although the discrete number of these tools is only about a third that of computer-based tools.

Interview participants clearly do not reserve or confine certain tools to particular design activities. While de Croock et al. (2002) report distinct ID tools to be used for specific design activities, following a prescribed design model (4C/ID), this may have to do with the perspective brought to that study. Stolterman et al. (2009), who studied design tools in use among interaction designers, and Gray et al. (2015) studying design judgments made by ID practitioners at work, both used an open perspective in which no presumption was made that design models determine the use of tools in a specific manner or particular order. The findings in this study conform to those, because—as practitioners report—their projects do not conform to a regular process and the tools they use may be applied to more than one part of a design process. For instance, participant 9I remarked “There is no order to it. You jump-in and you see what needs to be done and you do what needs to be done and you keep your eye on the goal all the time. That’s as far as like how to do a project”. Further, asked to make their own classification of tools, some respondents used process categories and others did not. Multiple respondents placed the same tool into different categories than did others. Participant 3C for instance classified his tools around his own job duties as an ID mentioning “communication”, “planning and mapping”, “content development”, “assembly”, and “training” as categories. Whereas participant 8H classified his ID tools (un-exclusively) into “Design”, “Develop”, “Deliver” and “Assess”. Participant 1A had a different approach by reporting specific design activities as categories of her reported ID tools, such as “Analysis and Scoping”, “Design”, “Support Learning”, “Development”, “Graphics production”, “Evaluation”, “Project management” and “Knowledge base”. The implication here is that designers are placing these tools at their service rather than that tools guide their design processes. It is telling that while designers listed tools that can be used step by step (like ADDIE as a conceptual process model), they did not list the kinds of automated design tools or expert systems that direct their activities. Stolterman et al. (2009) describe the major characteristics of what they call designerly tools as those that support the practitioners’ work without guiding it, are appreciated by practitioners each for different reasons, and are not used exclusively for specific, or individual, design activities. Our conclusion is that, regardless of what scholars may develop or recommend, practicing instructional designers choose and use designerly tools.

This evidence of a designerly approach to tools and their use is bolstered by the high frequency of interview statements describing instrumental judgment (Nelson and Stolterman 2012), a form of design judgment exercised to match processes and tools appropriately with the demands, global and specific, of a given design situation. While Dicks and Ives (2008) do not use the term design judgment when they report on the use of social and cognitive skills by ID practitioners in support of their daily ID practice, they are pointing to basic elements of the same construct. They point out that such tools are not prescribed in well-known design models (e.g., ADDIE), and are not being taught in ID classes.

While larger studies, preferably observational studies in the field, are called for if our field is to build a true understanding of how instructional design is practiced we posit, based on this study, that several ideas should be entertained by instructional design educators and scholars. First, educators need to focus on developing instrumental judgment in their students; this might entail an open-ended approach to selecting tools in their classes rather than specifying which tools (digital, analog and conceptual) their student must use, and promoting deep reflection in students regarding the efficacy of the tools they have selected and used. Scholars need to appreciate the skill required for instructional designers to do what they are already doing in selecting tools, matching them to the variable demands of the design situations they encounter, and using them flexibly for the properties best suited to their work instead of seeking to replace their designerly judgment with external guidance that ignores their legitimate skills. To do so, commitment to descriptive, versus evaluative, study of instructional design practice must increase among instructional design scholars.

Self-reported data may or may not reflect an accurate description of practitioners’ use or views on ID tools, although the exploratory nature of this study made self-reporting an efficient means to obtain preliminary insights. Despite the possibility of social desirability bias inherent in interviews (Grimm 2010), and in this case the possibility that respondents may have wanted the interviewer to see them in the best professional light possible, was addressed as much as possible by taking a non-evaluative focus in the recruitment message used and emphasizing the same message at the beginning of, and throughout, each interview.

As highlighted earlier in the problem statement section, IDT scholarship predominantly aims to measure the work of ID practitioners “against what academics think they should be doing” (Tracey and Boling 2014, p. 658). While this approach might have some value, it has limited our understanding of how instructional design actually occurs in situ. We believe this study adds to the evidence that ID practice, when studied closely, reveals a complex picture that bears further examination and deeper understanding if scholars are to attempt to support or improve it and educators are to prepare the most effective designers possible. Our descriptive study contributes to an understanding of ID practice from the view of the practitioners themselves, who talked freely about how they actually work and reported their tool use to us in an open-ended format. Their use of tools certainly shows some constraints and influences, often from the workplace and from their personal, professional preferences. It does not show that they are seeking or using tools created to scaffold or direct their designing in a predetermined way. Their ability to discuss the instrumental judgments they make regarding tools implies that this proclivity on their part is a valid feature of their practice, not ignorance of such tools or a willful refusal to use what is good for them. What good does it do scholars and educators in the field to appreciate this feature of ID practice? First, tool design for these practitioners can factor in their judgment and the flexibility with which they use tools, making the work of tool design both easier (less need to consider many fail-safe, or “idiot-proof” features), and more difficult (more need to consider how the tool might serve the designer instead of guiding the designer). Scholarly tools offered to the field might therefore represent wasted effort less often, but might—even better—contribute to advancement in the field robustly and affect positively the types of instructional design problems we can take on and the ability of our designers to flex with the nature of those problems.

Educators preparing instructional designers may also use the findings from studies of practice to refine both design briefs (assignments for project work) and the conditions they create for students to practice ID to reflect, and demand, the kind of designer-led activity that practice in the field requires. The result could be that ID students develop a strong sense of their instrumental judgment and confidence in their ability to use it, becoming practitioners who are not stymied or dismayed when a new design problem does not yield readily to the tools they have been taught to use. While we see that practicing designers do learn to exercise flexible use of tools, as educators it is a worthy goal to prepare students as ready to work effectively as we can make them. Every practice relies on the workplace to complete the education of new practitioners to some extent, but narrowing the gap would be a positive outcome from a realistic understanding of what practice in that workplace demands of our students when they enter it. For educators teaching practitioners already in the workplace, our instruction of them is enhanced in credibility and applicability when we recognize the realities of their practice.

Finally, an empirically established view of practitioners in our field behaving consistently with design theory allows us to draw on design theory when we hope, as we must, to improve our understanding of practice and contribution to its improvement.