1 Introduction

Digitalization has disrupted the global higher education system, and emerging digital technologies, such as the metaverse, next-generation virtual learning environments, and particularly generative AI, have brought new learning opportunities and have also forced us to reflect on learning design practice (Li et al., 2022a; Lodge et al., 2023). Yet at the same time, “education is inextricably influenced by the context in which it operates” (Bower, 2017, p. 2). Therefore, ensuring sustainable high-quality learning and teaching experiences in a changing digital world is a highly complex endeavour, and learning design for this digital world is a highly context-specific process. It requires learning design expertise, which involves a combination of technical skills and pedagogical knowledge, as well as an ability to apply that in specific contexts with a focus on specific cohorts of learners. Xie et al. (2021) have identified learning designers (or instructional designers) as essential change agents in developing “structured learning with online and digital resources as they have rich experience in preparing for quality online learning” (p. 332), which does not necessarily apply to discipline-based academic teachers. Similarly, there is an increasing emphasis on learner engagement in online environments, and how to design for that (Carroll et al., 2021). Quality learning design, which promotes learner engagement in online environments, relies on the application of in-depth pedagogical knowledge, for example about concepts such as Bloom’s taxonomy and constructive alignment (Biggs & Tang, 2011). In addition, it provides ways to address complex and contextual challenges of designing for online learning environments.

Bloom’s taxonomy (Bloom, 1956) and its extended version by Anderson and Krathwohl (2001) have been commonly used as a helpful planning and reflection tool for learning design. Biggs and Tang (2011) have highlighted the importance of ‘constructive alignment’ because educational learning activities and assessments not matching the intended learning outcomes is a commonly witnessed problem, which can negatively impact learning. In addition, different disciplines across different countries have different demands (Shahjahan et al., 2022; Whitsed & Green, 2016), which makes the learning design even more challenging, as there are different stakeholders to consider in different contexts. Shinde et al. (2018) have introduced what they call a fishbone diagram, or a ‘cause-and-effect’ diagram, as a tool “to identify the root cause of problems which represents the effect and the factors or causes influencing it” (p. 653), with a specific focus on student problems.

The fishbone diagram is a commonly used management tool for quality control in industry. It presents a “template for brainstorming possible causes of an effect” (Shinde et al., 2018, p. 653). Because of its nature of problem-solving and target aligning, a number of researchers have begun to explore the fishbone diagram as a way to address complex educational problems (Puspita et al., 2023; Shinde et al., 2018). Potential factors influencing a ‘student problem’ are broadly grouped under four broad headings: resources, university, personal, and academics. To each of these, subsets of factors are attached as ‘fishbones’, for example ‘curriculum design’, ‘assignments’, ‘study pressure’, ‘class mates’, etc. (Shinde et al., 2018, p. 657). In this study, we use this fishbone diagram as a basis for a digital learning design process.

The novelty of our study lies in the way it develops a learning design method to ensure constructive alignment. Specifically, this study integrates the fishbone diagram in a digital learning design process based on Bloom’s taxonomy for the digital world (Grantham, 2017), which includes seven educational objectives (remember, understand, apply, analyze, evaluate, create and share) (Abram, 2014; Churches, 2008), and it draws on a hands-on curriculum development workshop created at University College London, which uses the idea activity cards as a critical reflection tool (Young & Perović, 2020). Critical reflections are discussed, based on a research-informed post-design evaluation of two pilot courses. The key findings contribute to digital educational development and teacher professional development by proposing a practical and easy to adapt learning design method. Future studies will be conducted to further explore its application in other contexts and make continuous improvements.

2 Related studies

2.1 Learning design and design for learning

Learning design is a rapidly evolving field in dynamic higher education contexts, and learning design methods are therefore constantly evolving as well. Mor et al. (2015) have noted that learning design, or as some prefer to call it design for learning (Kickbusch et al., 2022; Laurillard, 2013), is an emerging field of educational research and practice. The difference is significant, as the concept of learning design has traditionally been associated with technology-enhanced learning (Conole & Fill, 2005; Salmon, 2013) and instructional design (Brown & Green, 2019), while design for learning shifts the focus firmly to the ‘design’ part of design for learning. Mor et al. (2015) further stress the dual nature of learning design, as “both a creative practice and a rigorous inquiry” (p. 221), reinforcing the idea of learning design as both deliberative and creative.

Within the context of learning design and instructional design focused literature, there has often been an emphasis on both tools and processes for learning design, which are implicitly, and sometimes explicitly, targeted at learning designers. For example, Salmon’s e-tivities present a set of activities for active and interactive online learning (Salmon, 2013), while the widely applied ADDIE model (Analysis, Design, Development, Implementation, and Evaluation) presents a process of stages instructional designers can use in their design for online educational environments (Spatioti et al., 2022). Furthermore, many learning design methods are based on a set of fundamental principles. For example, Tanis (2020) has identified the following seven principles, based on Chickering and Gamson (1987): “Faculty-student communication and collaboration; student–student communication and collaboration; active learning techniques; prompt feedback; appropriate time for tasks; high performance expectations; and respect for diverse learning preferences” (p. 2). These then inform specific learning design approaches or methods, such as personalized learning, peer learning, smart learning, self-regulated learning, independent learning, learner-centered learning, facilitative learning, collaborative learning, free-choice learning, and lifelong learning (Kavashev, 2024).

It is however important to distinguish between learning design and instructional design as well. Instructional design has a much longer history, while the field of learning design emerged at the beginning of the twenty-first century (Dobozy & Cameron, 2018). “Learning design is now acknowledged as a complex and integrated process, which includes stages of planning, designing, orchestrating and running of sequenced teaching and learning activities” (Dobozy & Cameron, 2018, p. i). This therefore requires considerable expertise that takes time to develop, in much the same way as disciplinary expertise takes time to develop. As Dobozy and Cameron (2018) have noted, “the pedagogical sequencing of online learning activities that are interactive and engaging and clearly aligned with contemporary learning theories, can be complex and especially demanding for academics new to learning design thinking and practice (p. i).” The fishbone diagram and approach that this study reports on directly addresses this issue in that it aims to provide structure for teachers to assist them in their learning design. It does so by providing a template for reflection to explore potential causes for particular educational issues or ‘problems’ in a specific context. These reflections can then be used to inform the learning design for that context. The ultimate aim is of course to develop learning designer expertise in teachers, rather than learning design expertise, but this is necessarily a longer-term goal, so in the meantime, the fishbone approach provides useful scaffolding.

With regards to learning design Goodyear et al. (2021) have noted that “what students actually do may differ considerably from what their teachers think they are doing or what their teacher intend them to do” (p. 446). This is a crucial recognition that applies not only to teachers but also to learning designers, for it suggests that you can design activities for learning, but the extent to which those activities actually lead to learning is highly context-dependent and dependent on a range of factors that may be beyond the learning designer’s control. Recognising this limitation is an important part of learning designer expertise, as it directly affects how they might design for learning. The fishbone diagram, or what we will call the fishbone digital learning design process in our case study below, allows learning designers to explicitly identify, distinguish, and reflect upon factors that they can influence, and those that are out of their control.

3 The case study

To engage with these issues, we conducted a case study (Yin, 2018). We first introduce the fishbone digital learning design process and then evaluate its value in aligning educational objectives for quality education. This case study included interviews and surveys with 20 student participants and critical reflections of four teacher participants (Fook, 2011), to investigate two research questions: 1. How can the fishbone digital learning design process be used to better align learning processes with educational objectives? 2. What is the impact of the fishbone digital learning design process on course experiences?

3.1 Case setting and participants

The selected case is an international university in China, namely University X, which is one of the largest independent Sino-foreign cooperative universities approved by the Ministry of Education in China, providing over 100 degree programmes across multiple disciplines. In response to the changing demands of modern society, University X creates many new programmes to provide new learning opportunities and prepare future-oriented professionals. Effective learning design is a critical priority in providing high-quality education through these new programmes. However, as mentioned above, there is no universal learning design method, and studies into digital learning design are varied in different contexts (Bower, 2017).

The MSc Digital Education postgraduate program within the Academy of Future Education (AoFE) was launched in 2022. It aims to cultivate learning designers with global vision and digital competence in an educational field frequently disrupted by emerging technologies. This new program is a good fit for our case study for two main reasons. First, as a newly launched program, it provides a university-approved program and course specifications, while the learning design needs to be further specified by course instructors. Therefore, the course instructor has considerable motivation to explore a practical digital learning design approach. Second, students enrolled in this program are expected to be future learning designers, seeking learning designer expertise, and therefore have sufficient motivation to participate in digital learning design method development and implementation as part of their learning.

Figure 1 shows the timeline of the FDLD development and application. One of the authors was the instructor for the two courses, who initiated the fishbone digital learning design (FDLD) method, and piloted learning design implementation from August to September 2022 during the course preparation period. The first cohort in the digital education program consisted of 20 master's students. We selected two core courses of the program for the case study: EDS431, Designing Digital Education Curriculum, and EDS433, Designing and Practicing Digital Education. EDS431 was delivered from September to December 2022, while EDS433 was delivered from February to May 2023.

Fig. 1
figure 1

Timeline of the FDLD development and application

Full size image

4 Research methods

This study employed a mixed-methods case study combining quantitative and qualitative approaches to investigate how the fishbone digital learning design process can be used to support constructive alignment. The research methods were selected to provide a richer and more in-depth picture of the case (Yin, 2018). Moreover, we collected data from multiple sources to mitigate the potential bias from the insider role of the author being the course instructor. On the other hand, the insider role provides sufficient opportunities for site observations and full access to empirical data, such as learning and teaching materials and learning activity logs. We followed the university ethics guidance and received approval in June 2022 (Appendix A) for data collection with teachers and students. Participant information and consent forms were signed by participants before the start of the study.

Table 1 shows the overview of the data sources. The course instructor further kept regular observation notes and a reflective journal from August 2022 to May 2023. These documents recorded three types of information: (a) consultations with senior faculty members about the design and development of the course specification, lesson plans and assessment task sheets; (b) informal and formal peer feedback on learning design drafts through peer support meetings and university moderation procedures; and (c) weekly reflection notes on student feedback. The university registry department organised student course questionnaires at the end of each semester during December 2022 and May 2023. The anonymised student responses were collected for instructor reflection and data analysis.

Table 1 Overview of the data source and collection approaches
Full size table

Apart from the university course questionnaires (item 3 in Table 1), the instructor administrated research questionnaires (item 4 in Table 1) in November 2022 and April 2023 to collect students' self-reported perceptions of their learning experiences during the semester. The questionnaire items were adapted from learning engagement scales by Deng et al. (2020), learner autonomy scales by Alowayr and Al-Azawei (2021) and Lakhal et al. (2013), and a learning performance scale by Mohammadyari and Singh (2015). Semi-structured interviews were conducted with four students who indicated their willingness to be interviewed via their online questionnaire responses. The university provides a centralised Moodle-based learning management system (LMS), called Learning Mall. The LMS stores all the learning and teaching materials, such as teaching notes, slides, reading materials, multimedia content, and online learning activities. In addition, the LMS allowed us to keep a detailed log record of all the online behaviours of teachers and students, such as login times, online learning activity attempts, completion status, online resources visit frequency, and posts.

This study employed a thematic analysis (Clarke, 2006) method to analyse the core data, including the instructor’s observation notes, reflective journals, student feedback on open questions in the course and research questionnaires, and student interviews. Descriptive statistics (Duesbery & Twyman, 2020) and content analysis (Frey, 2018) were used to triangulate the validation and trustworthiness of the core data. For example, the LMS log shows that the instructor's creation of a specific online learning activity can be used to verify if the planned learning design has been implemented in practice. The students' online learning behaviours can be cross-checked with their self-reported surveys and interviews.

5 Results

5.1 The fishbone digital learning design (FDLD)

To address the research question of how to align learning processes with educational objectives in digital education, we conceptualised five critical stages of the fishbone digital learning design (FDLD) cycle (Fig. 2). The evaluation and reflection stage has two dimensions: (a) the pre-design course evaluation and reflection; and (b) the post-design course evaluation and reflection. The FDLD cycle starts with the pre-design course evaluation and reflection, followed by setting milestones, anchoring educational objectives, aligning pedagogies and technologies, mapping learning processes, and the post-design course evaluation and reflection. As a sustainable development cycle, the endpoint is also the starting point for a new round of learning design.

Fig. 2
figure 2

The fishbone digital learning design (FDLD) cycle

Full size image

5.2 Stage 1. Evaluation and reflection: (a) Pre-design course evaluation and reflection

As Fig. 2 shows, the first stage in the FDLD cycle is “Evaluation and Reflection”. As aforementioned, this stage is special because it includes two parts. This section introduces the first part: pre-design course evaluation and reflection. It is essential to evaluate and reflect on the prior learning design of the target course. However, in this case study, the selected courses came from a new program with no prior design to examine. Therefore, the instructor checked the first draft of the course specification. Table 2 shows the educational objectives and example processes used for the pre-design course evaluation and reflection. Seven levels of educational objectives were adopted from different references. The first six levels (remember, understand, apply, analyze, evaluate, and create) originated from Anderson and Krathwohl’s (2001) revision of Bloom’s (1956) taxonomy. The example processes were extracted from Bower (2017), referencing Churches’ (2008) digital processes. The seventh educational objective, ‘share’, was adopted from Abram’s (2014) digital taxonomy reference table.

Table 2 Overview of educational objectives and example process with relevant references
Full size table

To better facilitate the course instructor for the pre-design evaluation and reflection, the FDLD method provides a template with an empty spider diagram. The course instructor can remark on the 0–5 Likert-scale scores, which measure the degree of each educational objective’s importance within the course (0 = low, 5 = high). In other words, it illustrates how much this course aims to help students achieve different levels of learning. Hereafter, we call this the educational objective weight. After remarking on the scores for each educational objective, the instructor can link the remark points to complete the radar chart and have an overview of the objective weight.

In our case study, as the first stage of the FDLD cycle in Fig. 2, the course instructor used the pre-design evaluation and reflection template to self-report the original educational objective weight based on the information in the course specification. Figure 3 shows the evaluation result of the course EDS431 as an example. The basic course information is recorded on the left, including the course name, program name, teacher name, the required learning hours, the course credit in the program, and the course learning outcomes. The seven coloured circles represent the seven educational objectives. Before using FDLD, the course had a primary focus on remembering and understanding (score = 4), followed by applying, evaluating and creating (score = 2), and analysing (score = 1). The seventh educational objective, ‘share’, is missing (score = 0). The instructor consulted senior colleagues internally and externally and received valuable feedback, based on those scores. For example, one of the senior colleagues commented,

According to the QAA (Quality Assurance Agency) standard, remembering and understanding are fundamental but should not be the focus of an FHEQ (According to the QAA (Quality Assurance Agency) standard) level 7 course. I would suggest you consider increasing learning activities that could better support students in achieving higher-level educational objectives, such as applying their knowledge in solving real-world problems.

Fig. 3
figure 3

EDS431 pre-design course evaluation form

Full size image

The pre-design course evaluation and peer feedback helped the instructor reflect on the improvement areas and inspired learning design ideas.

5.3 Stage 2. Setting milestones

According to Fig. 2, the second stage is “Setting Milestones”. Figure 4 shows a template provided by the FDLD method to better facilitate instructors to complete the second stage in the FDLD cycle. A milestone represents an important stage in that the students meet the expected educational objectives toward the completion of the course study. The milestones can be considered as time stamps for periodic learning outcomes to be assessed. Setting the milestones enables the instructors to think about the key question: when to assess and how (formative or summative)? For example, in our case study, the instructor sets seven milestones for EDS431. Milestone 1 is at the end of week 2. Milestone 2 is at the end of week 4, and so on. Students’ learning outcomes at milestones 1–3 and 5–6 will be assessed by formative assessment activities while learning outcomes at milestones 4 and 7 will be assessed by two summative assessment activities. It is important to note at this stage that what needs to be assessed was broadly identified as per usual constructive alignment practice, but detailed assessment design was beyond the scope of this particular case study, which is further explained in the limitations section. Although the template provides seven milestones, instructors can decide if to keep all the milestones or delete any of them for different courses. Moreover, instructors can define the period of each milestone according to the holistic learning design needs. For example, the instructor can set only two milestones for a course, while Milestone 1 is at the end of 3 weeks and Milestone 2 is by the end of 8 weeks to achieve higher level educational objectives. By using the milestone template, instructors can easily delete unnecessary milestones and edit the text of the remaining milestones to indicate the specific stage and time.

Fig. 4
figure 4

Fishbone milestone setting template

Full size image

5.4 Stage 3. Anchoring educational objectives

After setting the milestones, the second stage of the FDLD cycle was completed. In the third stage “anchoring educational objectives”, instructors can put the ‘flesh’ on the fishbone. The ‘flesh’ includes two parts: (a) the educational objectives to be achieved by every milestone point; and (b) the relevant pedagogies and digital tools which can help students to achieve the educational objectives. Instructors can put the coloured circles on the fishbone template. Multiple circles can be overlayed if there are multiple educational objectives for students to achieve at a certain point. The visible circle area represents the weight of the educational objective for that milestone. In a face-to-face workshop, the circles are movable stickers for instructors to paste on the paper template and make quick changes.

Figure 5 shows the educational objectives anchoring for EDS431 as an example. As an initial pilot, the instructor used PPT as the learning design tool to copy and paste the circles on the e-copy of the template. The course learning outcomes were broken down to align with the milestone settings. For instance, to achieve learning outcome A: Understand and apply a range of approaches to the design of online and offline courses, students need to achieve the educational objectives of Understand and Remember at an early stage. Therefore, the instructor anchors the purple circle (Remember) and dark blue circle (Understand) to the fishbone that links to the first milestone. Considering EDS431 is a masters-level course, the instructor decided to focus on Understand rather than Remember. Therefore, the dark blue circle is above the purple cycle.

Fig. 5
figure 5

Example of the educational objectives anchoring

Full size image

5.5 Stage 4. Aligning pedagogies and technologies

In the fourth stage in the FDLD cycle “aligning pedagogies and technologies”, the instructors can use the alignment cards to write down the selected pedagogies and digital tools. As shown in Fig. 6, the instructor selected research-led learning (Zhang et al., 2017) pedagogy to help students achieve the educational objective, Evaluate. Digital tools like online peer review and online choice were selected to support the pedagogical approaches. Instructors can write down different pedagogies and digital tools on different cards aligned to the same-coloured educational objective circles.

Fig. 6
figure 6

Example of the alignment cards for weeks 7–8 of EDS431

Full size image

5.6 Stage 5. Mapping learning processes

In the fifth stage of the FDLD cycle, "mapping learning processes", instructors meticulously specify the learning processes to align with the anchored educational objectives for each milestone, selected pedagogies, and digital tools. A detailed assessment design is integrated into this stage to support measuring learning outcomes. The FDLD process does not prescribe a rigid template for learning process mapping, allowing for inclusiveness and encouraging instructors' creativity.

Figure 7 presents an example of learning process mapping for EDS431, including the information on the timeline, pedagogical approaches, technologies described in the original section and the detailed formative and summative assessment design. The synchronous and asynchronous learning description of lecture and tutorial sessions outlines the assessments used to measure student progress towards achieving each educational objective. Both traditional and innovative assessment tools are utilized, taking into account the nature of the learning activities and the objectives being assessed. The assessments are designed to provide formative feedback to students, enabling them to adjust their learning strategies. Summative assessments are also included to evaluate student performance at the end of a milestone or the course. For instance, in tutorial 1 of week 2, where students are expected to interpret key concepts of Education 4.0 using an online glossary, the digital escape room 1 quizzes are designed to assess their understanding (Remember). In weeks 7–8, where research-led learning is employed, the “Curriculum Plan Critique” mid-term assessment measures students’ first half-semester learning outcomes through an essay-type coursework assignment.

Fig. 7
figure 7

EDS431 learning process mapping

Full size image

By integrating a detailed assessment design into the learning process mapping, instructors can more accurately measure the achievement of learning objectives. This iterative approach ensures that assessments are aligned with educational objectives and learning activities, promoting student success and continuous improvement in instructional practice.

5.7 Round two stage 1. Evaluation and reflection: (b) Post-design course evaluation and reflection

The FDLD process emphasizes the initial design and the ongoing assessment, feedback, and refinement of the learning experience. While completing the five stages of FDLD lays the foundation for an effective learning design, engaging in post-design evaluation and reflection is crucial to ensure continuous improvement. This post-design stage serves as a launchpad for a second round of FDLD, again highlighting the iterative and adaptive nature of the method.

In our case study, the instructor conducted a post-design evaluation by self-reporting scores for each educational objective on the spider diagram. This process allowed the instructor to assess the alignment between the designed learning activities and the intended educational objectives. For instance, the instructor observed that while the level of Understand remained consistent (score = 4), the level of Remember decreased slightly (from score = 4 to score = 3). Conversely, scores for objectives such as Apply, Analyze, Evaluate, Create, and Share increased, indicating areas of success and areas requiring further attention (Fig. 8).

Fig. 8
figure 8

EDS431 post-design course evaluation form

Full size image

To provide more comprehensive guidance for instructional improvement, we recommend the following assessment and feedback mechanisms:

  1. 1.

    Multi-source Feedback Collection

    Besides self-reported scores, instructors can seek feedback from students, peers, and subject matter experts. Student surveys, focus groups, and individual interviews can yield valuable insights into the learning experience. Peer reviews and expert consultations can provide an external perspective on the design and its effectiveness.

  2. 2.

    Quantitative and Qualitative Analysis

    Both quantitative and qualitative data should be analyzed to gain a holistic understanding of the learning outcomes. Quantitative scores such as those used in the spider diagram can indicate overall trends. Qualitative feedback, such as student comments and instructor reflections, can provide nuanced insights into specific aspects of the learning design.

  3. 3.

    Alignment Analysis

    Instructors should analyze the alignment between the designed learning activities, assessments, and the intended educational objectives. Discrepancies may indicate areas requiring refinement or redesign.

  4. 4.

    Iterative Refinement

    Based on the assessment and feedback, instructors should make iterative refinements to the learning design. This may involve modifying learning activities, assessments, or instructional strategies to achieve the educational objectives better.

6 Discussion

6.1 Evaluating the impact of the FDLD process

To address the second research question about the impact of the FDLD process, we compared the learning design results and evaluated the course experiences. Two themes of the impact emerged.

6.2 Impact 1. Visualize the educational objectives and enrich the learning activities

The first impact is that the FDLD process makes the educational objective more visible and encourages instructors to design different learning activities to meet the needs of achieving different educational objectives. Specifically, the FDLD process visualizes educational objectives with different colours and embeds these visual elements in the learning design templates, which reminds instructors to consider the variety of learning goals to ensure a quality educational process. The visual elements include the spider diagram for pre-design and post-design evaluation in stage 1, the movable colour circles for anchoring educational objectives in stage 3, the colour cards for pedagogical and technological alignment in stage 4, and the colour keywords in stage 5 for the learning process mapping.

For example, in our case study, the spider diagram in the course evaluation stage visualizes the weight of each educational objective via the instructor’s self-reported scores. Table 3 compares the educational objectives scores in the pre-design and post-design course evaluations for the two pilot courses: EDS431 and EDS433. The pre-design scores for the two courses were the same because the instructor perceived the same level of educational goals in the two courses’ specifications. However, the post-design scores were different. Apart from the educational objectives of Apply and Understand, all the other objectives had higher scores in EDS433 than in EDS431. The instructor reflected:

The FDLD process keeps reminding me to consider the value of different educational objectives. If students only achieve the foundational objectives like Remember and Understand, it is hard for them to develop higher-order thinking and problem-solving skills for better employability. In the first semester of the course EDS431, I reduced the number of the purple and dark blue circles and increased the number of other circles at stage 3 while anchoring the educational objectives. As a result, these actions lead to the changes in the post-design evaluation result. Students’ positive feedback and satisfactory performance encouraged me to keep increasing the weight of the higher level educational objectives for EDS433 in the second semester.

Table 3 Descriptive log statistics and course feedback scores of the two pilot courses
Full size table

Moreover, the FDLD process can not only visualize the educational objectives but also enriches the learning activities. Table 4 shows the descriptive statistics of the two courses. Both courses had more than 100 online learning activities including files, URLs, quizzes, forums, glossaries, interactive videos, digital escape rooms, and assignments. The reasons are twofold. First, the instructor intended to be inclusive and ensure that a range of educational objectives could be addressed by relevant learning activities in different learning periods. For example, the weighing score of the objective Apply for EDS431 is 4, which is quite high. To scaffold the students' achieving this highly weighted educational objective with each milestone, the instructor created 13 digital escape room activities throughout the semester. Students could practice applying their knowledge in each escape room activity every week. Second, the large number of online activities was also relevant to the nature of the two courses that aimed to demonstrate the use of different online learning activities for digital education. Therefore, the number of online learning activities in this case study does not reflect the general needs of every course and is quite context-specific.

Table 4 Descriptive log statistics and course feedback scores of the two pilot courses
Full size table

Additionally, Table 4 shows that EDS433 has more online activities than EDS431. This result is related to the evaluation scores in Table 3. As aforementioned, the instructor increased the scores in EDS433 for all the higher-level educational objectives, which further enriched the learning activities. For example, to help students achieve the educational objective Share, the instructor created 10 online forums in EDS431 (Share = 2) and 17 online forums in EDS433 (Share = 4).

6.3 Impact 2. Influencing learner engagement, autonomy, and performance

Using the FDLD process can influence learner engagement, autonomy, and performance. As shown in Table 4, the weekly activity attempts per student in EDS431 (72) and EDS433 (61) are relatively high considering the small number of students (n = 20). However, EDS433 has a lower weekly activity attempt number per student. To further investigate the in-depth reason, we used NVivo 20 to do data analysis and the qualitative results indicated two potential reasons. First, students spent a longer time creating each forum post than attempting a simple quiz question. Although the number of attempts was less, the level of deep learning might have been higher. For example, student A commented on the university course questionnaire:

Because this module is a flipped learning way, so I have to spend time on its pre–reading. It forces me to understand and organise this content for the lecture, and I am accustomed to Xmind in daily life. I think I began more active than before; I always learn in a passive way, only listening and doing nothing. I know it is not enough, but I think it is a good opportunity to improve myself. The teacher always pays much attention to our questions in the forum and always gives us feedback kindly. Sometimes I think my own performance is bad and shamed, but the teacher has positive feedback and never blames me. I was afraid of communicating with teachers, but now I begin to perceive it is not a horrible thing like before.

Student B commented:

The assessments are well-designed because by finishing the first one, we can sort of understand the expectations of the second one in a more authentic way. Besides, there is clear guidance to the completion of our assignments. The final presentation is a good simulation of presenting our learning design to peers because we can experience what the possible users are interested in as feedback for future learning design.

Second, in EDS431, a weekly online quiz was used as the formative assessment to help students achieve educational objectives such as Remember and Understand in the first semester. The nature of the online quiz is for students to repeat attempts until they are satisfied with the quiz results, which thus indicates their learning progress. Therefore, the number of online activity attempts in EDS431 is greater than in EDS433.

To further evaluate the impact of the FDLD process on the course experiences, we used the software Smart PLS 4.0 to analyse the students' self-reported questionnaires. Appendix B lists the descriptive statistics of the survey scale items that aim to measure student self-reported learning engagement (ENG), learning autonomy (AUT) and learning performance (PF). The mean value for EDS431 ranged from 3.05 to 4.15, while EDS433 ranged from 2.895 to 4.158. The standard deviation (SD) for EDS431 ranged from 0.707 to 1.005, while for EDS433, it ranged from 0.909 to 1.372. Both are acceptable, but extreme values occur more frequently in EDS433’s data. Overall, student perceptions of the learning experiences and performance were positive (Mean > 3, ranging from 1–5) in the two courses.

6.4 Practical implications

The FDLD process offers several practical implications for instructors seeking to improve teaching and learning outcomes in higher education. While traditional learning design methods provide valuable frameworks, FDLD offers a more comprehensive and iterative approach better suited to higher education's dynamic nature.

Specifically, previous design methods, such as the ADDIE model highlighted by Spatioti et al. (2022), emphasize a linear process that, while structured, may lack the flexibility and iterative nature required for contemporary learning contexts. FDLD's emphasis on alignment, visualization, and context-sensitivity enables instructors to develop learning designs that are tailored to specific educational goals and objectives. The use of visual elements, such as coloured circles and cards, makes these objectives more visible, prompting instructors to design a diverse range of learning activities that are tailored to achieving various objectives. This approach not only enriches the learning experience but also has the potential to enhance learner engagement, autonomy, and performance.

On the other hand, FDLD adopts a holistic and iterative approach that aligns closely with the emerging field of learning design (Mor et al., 2015). This approach focuses on the "design" aspect, emphasizing learning design's creative and rigorous nature as both a deliberative and creative practice. Moreover, the FDLD process highlights the importance of setting milestones and breaking down large educational goals into actionable steps aligned with specific assessments. This ensures that learning objectives are measurable and attainable, providing instructors clear guidance on assessing student progress and achievement.

More importantly, FDLD addresses the complexity of learning design, particularly for academics new to the field. It provides a structured template for reflecting and exploring educational issues in a specific context, allowing teachers to develop their learning design expertise over time. This scaffolding process can lead to more effective learning designs. Additionally, FDLD recognizes the limitations of learning design (Goodyear et al., 2021) and fosters a more context-sensitive and adaptive approach. It enables designers to explicitly identify and reflect upon factors they can influence and those beyond their control. FDLD can help instructors make informed decisions about their learning designs, considering their students' unique characteristics, teaching content and available resources.

Finally, the FDLD process supports teachers' professional development by promoting exploring and applying new teaching methods and technologies. This ongoing development enhances teaching capabilities and improves professional competence, enabling instructors to stay abreast of the latest trends and best practices in higher education.

6.5 Limitations and future development

Our study has several limitations that future studies can address for further improvements. First, constructive alignment encompasses both the alignment of educational objectives and assessment. However, this study has mainly focused on the alignment with educational objectives. Further research is needed to investigate the alignment with assessment. Specifically, the assessment design deserves a separate position in the alignment card. In the current FDLD version, the fishbone diagram shows clearly when and what to assess, yet it does not show how to assess, especially with digital tools. To align the assessments with the objectives and learning activities (Biggs & Tang, 2011), the alignment cards could add one more part: assessment. In this part, designers would be able to select assessment methods and digital tools regarding their pedagogical value (Li et al., 2022b). Again, this is especially important in the context of the changes brought about by GenAI. As part of the learning process mapping, it would be helpful if more assessment details, such as assessment criteria, were included in the Excel spreadsheet. In recognition of some of the limitations noted in designing assessments, particularly in strengthening students’ agency and autonomy, a modified assessment system such as Specification Grading forms will be included as part of the continuous improvement plan for this project. Furthermore, the advantages and disadvantages of pedagogy and digital tools could help designers make better decisions during the learning design.

Second, the selected courses were new; therefore, not much data was available for evaluation and reflection at the first stage of the FDLD process. According to outcome-based education (OBE) (Spady, 1994), course evaluation is vital for instructors to determine if the objectives are aligned with the teaching and learning activities and assessment. However, like OBE, the FDLD process largely relies on evaluations that are subjective. Future studies are needed to develop more objective tests and measurements to better facilitate the course evaluation process. Additionally, instructors might need extra help to set milestones. Ideal milestone setting requires instructors to match time stamps with periodic learning outcome assessments. There are dozens of verbs (intended learning outcomes) in Bloom’s taxonomy, yet the taxonomy lacks a systematic rationale (Morshead, 1965). This has led to dozens of stickers with different colours and meanings, making the fishbone milestone setting quite complicated. Moreover, instructors need to construct the meaning of the innovative approaches for long-term development (Li et al., 2022c).

Regarding the generalizability of the case study, it focuses primarily on a digital education programme within a specific context. While the results provide valuable insights into the application of FDLD in this particular setting, conducting case studies across various disciplines, educational stages, and cultural backgrounds would further validate the method's universality and effectiveness. Different disciplines have unique learning objectives and requirements, and applying FDLD across disciplines could reveal how the method adapts to and enhances learning experiences in diverse contexts. Similarly, testing FDLD at different educational stages, from primary to higher education, would demonstrate its applicability across the education spectrum. Considering cultural backgrounds is crucial as learning preferences and practices vary among different cultures. Exploring the use of FDLD in different cultural settings could reveal how the method needs to be adapted to suit specific cultural norms and values.

Finally, the current version of the FDLD focuses on the cognitive domain of Bloom’s taxonomy. Therefore, it is limited in addressing the other domains like the affective domain (the awareness and growth in attitudes, emotions, and feelings) and the psychomotor domain (the change or development in behaviours or skills) (Simpson, 1972). To enhance the effectiveness and comprehensiveness of FDLD, future research should aim to develop a version of the framework that can accommodate objectives from all three domains of learning: cognitive, affective, and psychomotor. This multidimensional approach would enable instructors to design learning experiences that are truly holistic, addressing not only intellectual growth but also emotional engagement and skill development. By doing so, FDLD has the potential to transform learning outcomes and prepare students for the complexities of real-world challenges.

7 Conclusion

The combination of the fishbone digital learning design method, as outlined in this paper, and Bloom’s taxonomy provides teachers and aspiring learning designers with a strong structure to scaffold their practice. This is important, as the process of acquiring learning designer expertise takes time and is honed through experience. Given the dynamic and ever-changing context of online learning, it should therefore be seen as a lifelong learning process. Developing learning designer expertise is in that sense comparable to developing discipline expertise (Czaplinski, 2020). However, it is important to note that there are numerous opportunities for further research and development in this field. This includes a deeper exploration of assessment and feedback mechanisms, assessment design alignment with educational objectives, refining course evaluation processes, and extending the FDLD method to encompass other domains (e.g., affective and skills) of learning outcomes across various disciplines, educational stages, and cultural backgrounds.

Yet, there has long been an expectation that teaching and learning design expertise of university teachers simply happens by osmosis. The fishbone digital learning design method therefore recognises the constraints on teachers of developing learning design expertise and could be seen as an effective scaffolded ‘shortcut’ of sorts. This certainly does not remove the need for ongoing development of expertise in highly dynamic learning contexts, and the fast emergence and impact of GenAI only serves to reinforce this point. However, it provides teachers and aspiring learning designers with a solid base from which to continue to build their expertise. By embracing these opportunities for further research and refinement, we can continually enhance the effectiveness of the fishbone digital learning design method and better support teachers and learning designers in their craft.