Keywords

Introduction

Open, Distance, and Digital Education (ODDE) is a mechanism to address the various challenges being faced by the education sector. The need for a flexible, agile, and resilient education system became more evident when the COVID-19 pandemic affected the whole world and forced all educational institutions to immediately shift to remote instruction just to ensure that learning would continue for their students.

The disruptions in education, however, are not a new challenge for the education sector. Discontinuance of learning also happens when students and teachers are prevented from going to school because of natural calamities like typhoons and floods, disasters like earthquake and volcanic eruptions, and even man-caused circumstances like the disruption in the peace and order situation in the area.

ODDE is also a viable strategy which can position higher education institutions to respond effectively to the projected demand for higher education estimated to reach over 414 million in 2030 (ICDE, 2015) as well as contribute to the goal of making available inclusive lifelong learning opportunities as demanded by the fast-changing world of work. The full adoption of ODDE as a system or a component of the system of learning, however, can be hindered by the still lingering perception that this form of education is of lower quality compared with the residential or conventional education despite research results showing otherwise. Asare (2014), for instance, argued that the focus should not be the mode of delivery but rather how learners are equipped with relevant knowledge and skills to become functional citizens which can be achieved even in the distance mode of education. Also, a publication released by the US Department of Education (2010) presenting the results of a systematic review of literature from 1996–2008 showed that “students who took all or part of their course online performed better, on average, than those taking the same course through traditional face-to-face instruction” (p. x1v). Shachar and Neumann (2010) also validated this finding in their research which also showed that “that DL [distance learning] students outperformed their traditional counterparts” (p. 322).

This perception of lower quality can be traced to as far back as the pioneering days of distance education or the correspondence system of learning which can be due to the lack of social education usually associated with what students can get when studying on campus (Tait, 2008). Citing de Salvo (2002), Tait (2008) further explained that a student’s university career is “more than the acquisition of knowledge” (p. 86) but being a “man among men” (p. 86) which can result from joining university extracurricular activities like debating society, football, and the likes.

Over the years, the implementation of distance education has evolved as influenced by the rapid advancement in the information and communications technologies (ICT). Most universities engaged in this mode of delivering instruction resorting to online learning or technology-enabled teaching and learning which was also seen to address the concern about the social education of the learners. The potential of ODDE was also implied by the Council on Higher Education (CHE), Pretoria (2014), that “there seems to be a widespread assumption that education mediated by means of ICT-supported methods can improve the quality of educational provision in developing countries, not least in institutions of higher learning” (p. 1).

The stigma of being of lower quality attached to online learning (Hodges, Moore, Lockee, Trust, & Bond, 2020) could be attributed to that observation that “students in online environments tend to feel more confused, isolated, and frustrated, and as a result their learning effectiveness and satisfaction can be reduced” (Markova, Glazkova, & Zaborova, 2017). Despite many research that aimed to address this issue of quality in ODDE and while results of these studies showed that the physical separation between the teacher and the students is not a major factor that determines quality education, the perception remains.

Two questions are forthcoming: Where are the possible gaps? And what are they?

Sloan Consortium (2002) observed that “despite research from respectable educators that points toward the positive effects of online learning, many still say that more sound studies still need to be conducted to measure and document the most effective kinds of online teaching” (p. 4) which points to the type of research that were conducted to address the concern. Markova et al. (2017), on the other hand, pushed for certain quality indicators to “be established to ensure high quality standards in distance tertiary education” (p. 686). Chao, Saj, and Tessier (2006) pointed to the methodology being used and forwarded the following observation:

Despite efforts in defining and examining quality issues concerning online courses, a systematic, formative methodology to measure and ensure quality is lacking. The most common tools for gauging quality are surveys and course evaluations in which instructors, learners, or sometimes administrators provide their perceptions, opinions, or experiences. Data collected from surveys or course evaluations only touch on some aspects of a course’s quality—mostly issues related to teaching and learning, such as how an instructor performs in class or how the learning experience affected learners. Often, aspects not obvious to faculty or learners are ignored… (p. 33)

The need to reframe how the quality of open distance and digital education is evaluated was also implied by Chao et al. (2006, p. 33) in terms of requiring “a comprehensive framework and appropriate guidelines as well as devise an instrument and method for measuring the hidden aspects of quality.” Council on Higher Education (CHE) (2014) is also of the perspective that “the question of the quality of educational delivery and support using ICTs requires much deeper analysis” (p. 2).

This chapter aims to address these implied quality determination and evaluation gaps in ODDE with the hope of contributing toward strengthening its position as the foundation of a resilient and sustainable education system.

The QA Frameworks for ODDE: The Domains That Matter

Just like in the conventional mode of instruction, there are already many frameworks for QA in ODDE. Many of these QA Frameworks were developed by different organizations during the last 20 years which can be attributed to the rise of open and digital education as a result of continuous and increasing level of integration of modern ICTs, specifically the Internet, into the teaching and learning process during this time span. An evaluation of ten QA Frameworks developed by various organizations from different parts of the world from 2000 to 2019 shows the different domains or benchmarks of quality for ODDE (Table 1). Information presented in Table 1 is from multiple sources. The selection of the QA Frameworks evaluated in this chapter was based on the following criteria: (1) It was developed by an organization/agency involved with ODDE and is not a result of a study by a researcher or developed by a specific university for its use (adopted from Pedro & Kumar, 2020); (2) it was developed during the last 20 years to determine the trend; and (3) it can represent areas/geographical locations from different parts of the world.

Table 1 Different QA frameworks in ODDE (2000–2019)
Full size table

In general, there are some agreements as to what constitutes quality in ODDE as evidenced by the similarity of the domains identified across the frameworks. Most of the frameworks studied were supposed to guide the development of ODDE programs and courses as well as the evaluation for quality which could be the reason why the domains identified were more of the inputs by the academic institutions offering ODDE which include among others institutional mission, vision, policies, planning, infrastructure, the faculty and staff credentials and continuous development, and the learner support system. Further, the QA Frameworks for ODDE during the last 20 years showed consistency in the inclusion of domains pertaining to programs and courses as these represent the products and services offered by ODDE institutions, as appropriately described by the Canadian Association for Community Education (CACE, 2002) in the way they framed their QA Framework. Some QA Frameworks are specific in identifying programs (or academic programs, for that matter), and the specific aspect that is being looked into in the framework like program design (CHE, 2014), curriculum design (Asia-Pacific Economic Cooperation (APEC), 2019; EADTU, 2016), program design and curriculum development (AAOU, 2019), and program standards (Commonwealth of Learning, 2019). CHE (2014) also included administrative services, coordination, delivery, and evaluation as additional aspects for evaluating programs for quality. Other QA Frameworks subsumed the program/curriculum component in the other domains as in the cases of IHEP (International Higher Education Policy, 2000), CHE (2014), and EADTU (2016). In the IHEP Framework, evaluation and assessment benchmarks include program effectiveness assessment through the following: enrollment data; costs and successful/innovative uses of technology; and regular review of the intended learning outcomes to ensure clarity, utility, and appropriateness of the ICT use. CHE, on the other hand, has a recommended framework for curriculum development and evaluation which can serve as a standard for evaluating for quality while EADTU focused on curriculum design which was articulated in terms of flexibility, academic community involvement, knowledge and skills, and assessment procedures.

For the course quality domain, some observations which can be drawn from the ten QA Frameworks studied are the following.

Course Development

There is a consistent inclusion of the course component in all the frameworks studied which implies the common perspective that this aspect is important in determining the quality of ODDE. Course is a main quality domain in IHEP (2000), EADTU (2016), Online Learning Consortium (2017), and AAOU (2019) frameworks and is subsumed or implied in other domains based on the domain descriptions and in the indicators or evidence of quality.

Different terms or nomenclature and descriptions were used to refer to the course domain for quality like course design (EADTU), course delivery (EADTU), course development (IHEP, 2000), teaching and instructional practices (OLC), learning effectiveness (SLOAN), and student satisfaction (SLOAN). The CACE Framework included the course component in the evaluation of process (approaches to learning; instructional strategies; and assessment of learning) and inputs and resources (intended learning outcomes; curriculum content; teaching and learning materials; learning technologies; and technical design of the LMS).

A more comprehensive description was forwarded by EADTU (2016) for course evaluation which starts at looking at the rational progression from one course to the next; the relationship of the course with the curriculum and the overall program learning outcomes, the course content design and the student interaction with the course material, the detailed development of the course materials, and assessment; and course evaluation and the process of course approval. For OLC (2017), course design covers course overview and information, course technology and tools, design and layout, content and activities, interaction, assessment, and feedback (https://onlinelearningconsortium.org/consult/oscqr-course-design-review/). IHEP (2000) has a domain for course structure which focuses on the different stages of the course delivery, e.g., orientation about the course and technology requirements before starting the online program, learning outcomes, library resources, assignments, and faculty response. IHEP also has course development benchmark to cover design and delivery, as well as determining the technology to be used based on the learning outcomes.

COL (2019) has a specific domain for course materials development which looks at the desired characteristics of the learning materials to include the following: presentation in appropriate formats that allow easy access by learners; and coherence between learning materials and learning outcomes and course content and assessment. Learning materials should teach in a coherent way, engage learners, and promote development of problem-solving and critical thinking skills. Learning materials should also be evaluated and updated on a regular basis. The CACE QA Framework (2002) also included teaching and learning materials as one aspect under inputs and resources.

Course delivery was also articulated in the different QA Frameworks focusing on different aspects of this quality domain. OLC (2017) includes teaching and instructional practices domain which covers the digital classroom experience, course fundamentals (course design, accessibility, and continuous improvement), learning foundations (course learning outcomes, course content, and assignments), faculty engagement (the instructor role like providing effective feedback, use of tools within the LMS to facilitate the learning experience in an effective manner), and student engagement (how the course was designed to facilitate class discussion engagement, building community, and communication). IHEP (2000) covers teaching and learning quality domain which pertains to student-teacher and student-student interactions as facilitated by various technologies, and providing constructive and timely feedback to students’ submissions. CACE (2002) has approaches to learning, instructional strategies, and assessment of learning as components of the process domain. For EADTU (2016), the specific domain for course delivery includes the Virtual Learning Environment, the personal learning environments, and the other channels, such as social media, through which students receive their course materials or communicate with fellow learners and staff. The selected systems should be driven by both educational and technical requirements. The educational requirements include delivery of learning resources, facilities for online communication, and tools for assessment while the technical requirements refer to the reliability and security standards. EADTU also emphasized that the delivery system should be reviewed and monitored to ensure that it continues to meet the educational and technical requirements.

The COL QA Framework (2019), on the other hand, puts emphasis on student assessment, which can also be considered as part of course delivery. According to COL, assessment strategies should be effective, valid, and reliable, and with appropriate security and QA measures to ensure the integrity of the assessment process. COL also included the presence of a policy for student appeals on assessment results and that the turn-around times for such appeals should be clearly defined in the policy.

In some QA Frameworks, the course domain is also implied in the student experience or journey while taking the course which can also be a result of the convergence of other quality domains like learner support services, infrastructure, credentials and qualifications of faculty and staff, and institutional policies.

In general, the different QA frameworks studied presented the course domain to cover the life cycle of the course which starts from its identification as a component of the curricular program, the development and design of the course/course materials, how the course is delivered to the learners, and the evaluation for subsequent revision and further improvement.

Educational Technology

There is emphasis on the selection and use of appropriate teaching and learning technologies vis-a-vis the intended learning outcomes when evaluating courses in the ODDE system. EADTU, for instance, described the whole course delivery process to include the different technologies used like the virtual learning environment where the learners receive the instructional materials, communicate, or facilitate the different interactions that are critical to the learning process and the tools for assessment and other learning activities.

Program and Course Evaluation

Program and course evaluation is a domain of quality itself. There are QA Frameworks which specifically include program and course evaluation as domains of quality such as EADTU, CACE, and COL. SLOAN Consortium (2005) also included learning effectiveness with the following indicators: new knowledge generated; theories applied to the workplace, continuous feedback from different stakeholders; and student satisfaction and loyalty. CACE has a specific domain for quality outcomes which can be determined by the relevance to employment of the content skills and knowledge acquired from the course/program, and the recognition of the course credits and program credentials by other education institutions and employers locally and internationally. For student satisfaction, CACE proposed that this be derived from the following: course effectiveness or the achievement of personal learning goals; course efficiency or the best use of student finances, time and energy; student satisfaction with processes and practices; and adequacy of inputs/resources provided to the students.

Program and Course Evaluation in ODDE: Perspectives and Practices

The perspectives expressed by various authors imply a comprehensive view which can be adopted when it comes to program and course evaluation. While there was a general agreement that program and course evaluation principles in conventional instruction and distance education are generally similar, many authors were also quick to point the major differences which can be attributed to the basic attributes of ODDE: the openness to provide access to education opportunities to the nontraditional learners; the separation in time and space of the learners from the teachers and fellow learners; and the higher integration of ICT into the teaching and learning process to bridge the physical separation. These attributes also make program and course evaluation in ODDE more complex compared to conventional instruction.

The Kirkpatrick (1975) model of ODDE program and course evaluation looks at four levels: (1) reaction and planned action which gauge the participants’ satisfaction and studying how they intend to apply what has been learned during the course; (2) learning which shows what the participant has learned from the course; (3) workplace application which pertains to whether and how the participants apply what they have learned to their respective jobs; and (4) business results which look at the benefits gained by the organization in offering ODDE programs and courses and can be in economic terms and/or customer satisfaction.

Different authors are in agreement that evaluation of ODDE programs and courses is more complicated because of the basic characteristics and attributes of this system of instruction, which include the usually big number of enrollment in the programs and courses, the geographically and widely dispersed student body (CHE-South Africa, 2014), and the diversity in the students’ profile which may have implications in quality outcomes and students’ satisfaction.

Further, given the major role that modern ICTs play in the ODDE system, there may be that tendency to just focus on the technology per se instead of its impact on the teaching and learning (CHE, 2014).

CHE (2014) further emphasized that in evaluating ODDE programs and courses, the focus should not be the technology per se but the effect on teaching and learning as explained below:

The question of the quality of educational delivery and support using ICTs requires much deeper analysis. Simply throwing computers at higher education institutions is in no way a responsible manner in which to begin to address quality improvement. While the provision of ICT hardware and related supporting network infrastructure, improvement in the provision and reliability of Internet access and connectivity, and implementation of relevant software applications are clearly important, it is only when the improvement of teaching and learning is addressed that claims made for the educational potential of supporting ICTs can be confirmed or refuted. (p. 2)

The Openness Attribute

Being open in widening access and participation while providing reasonable care to ensure success (Zawacki-Richter et al., 2020), ODDE programs and courses would affect the structure, the aspects to be evaluated, success definition, indicators, and criteria, as well as the evaluators who will participate in the evaluation for quality process. The openness attribute implies the wide variation in the profile of the learners and the geographical distribution which impact on the learning design and authentic learning. Questions like what examples to use, what resources to refer to, the language to use (CHE, 2014), and developing collaborative learning activities and the subsequent grouping of learners to maximize the learner-learner interaction need to be carefully considered.

Alturkistani et al. (2020) also emphasized the need for a separate evaluation system for open education as in the case of MOOCs, which are ideal representations of ODDE because teaching and learning in MOOCs is very different from the conventional mode of instruction and even in an online course which is not open. The current practices in evaluating MOOCs for quality center on three aspects: learner; teaching; and the MOOC itself. For the learner, aspects of engagement, completion rate, satisfaction, peer interaction, learning outcomes and experience, and knowledge retention were the parameters being looked into. For the teaching aspect, the pedagogical practice of teaching in massive enrollment was given importance. And for the MOOC itself, comparison with other learning platforms, content and structure, implementation, and sustainability are considered as indicators of quality (Alturkistani et al., 2020). It should also be noted that the current evaluation methods for MOOCs consist of surveys, interviews (email and online focused group discussion), pretest/posttest, and the data gathered from the Learning Management System which can include attendance rate; completion of the different MOOCs components; quiz or assignment scores; learner activity; and discussion posts.

The Mode of Instructional Delivery

Because of the nature of ODDE, the programs and courses can accommodate various types of learners and learning contexts. ODDE programs should be structured in such a way that it can accommodate and be responsive to the needs of various learners and as such implies the involvement of a multidisciplinary team to design the learning experience appropriate to these learners. The mode of instructional delivery highlighting separation between the teacher and the learners also assumes the exacerbation of the transactional distance (Hodges et al., 2020). This implies that course evaluation for quality should look at teaching strategies or instructional practices which are considered to be critical to learning like the instructional dialogue. Instructional dialogue is the interaction between the teacher and the students facilitated by the ICTs. The consideration for evaluation is whether opportunities for such dialogue are built into the design of the courses.

The ICT Integration

ICT integration into the ODDE programs and courses is not simply “transferring face-to-face education materials on the virtual setting” (Tonbuloğlu & Gürol, 2016). Enabling or enhancing the teaching and learning process through technology integration can result in different learning design strategies and possible increase in the variables and indicators which should be considered when evaluating for quality (Benigno & Trentin, 2000). As articulated by various authors, technology contributes to the enrichment of the learning environment as it facilitates interactivity and asynchronicity (Benigno & Trentin, 2000), encourages cognitive and psychosocial development (Kerr & Hiltz, 1982, as cited by Benigno & Trentin, 2000), and helps develop metacognitive skills and organized thinking process (Henri, 1992 as cited by Benigno & Trentin, 2000).

Designing learning for ODDE often considers the integration of the various types of interactions: learner-learner; learner-teacher; learner-content; and learner-community of practice which has been recognized to promote authentic learning. The asynchronous component of most online courses provides space as well as opportunity for the learner to digest course content and engage in self-reflections which can be building blocks of the learning process. The asynchronous online discussions also allow learners to consider and appreciate the diverse perspectives about the lesson, and organize the information gathered to build their own learnings.

Recommendations for ODDE Program and Course Evaluation

Various authors forwarded some recommendations for effective program and course evaluation in ODDE.

Alturkistani et al. (2020) recommended the use of the design thinking approach for better quality and precision of data that will be gathered and at the same time provide general guidance especially in terms of performance indicators which should be considered in the evaluation process. Burns (2018), on the other hand, suggested the adoption of the instructional design process which “begins with the end in mind” (p. 150) or a “backward mapping evaluation” (p. 254) and which consists of three steps or answering three questions: Who is the audience of the evaluation and for what purpose would the evaluation results be used? What do they want to know? How the information will be gathered or collected? (https://elearningindustry.com/evaluating-your-online-learning-program-part-1). The same design for evaluation was also suggested by the US Department of Education (2008), who aside from recommending both formative and summative program and course evaluations also recommended that the process should begin “with a clear vision for the evaluation” (p. 49) which determines the following: “what you want the evaluation to accomplish and what questions you hope to answer, the most appropriate evaluation methods for meeting your goals and the budget to meet evaluation needs.”

The recommendations on the process of evaluation include the shift from formal evaluation of learning to gauging the students’ participation in group and learning activities (Benigno & Trentin, 2000), peer evaluation (Stewart & Kogan, 2015), and for the process to be continuing instead of being “episodic” (University of Toronto Center for Teaching Support and Innovation, https://teaching.utoronto.ca/teaching-support/curriculum-renewal/program-assessment/).

In terms of evaluation data, the University of Toronto’s Center for Teaching Support and Innovation (https://teaching.utoronto.ca/teaching-support/curriculum-renewal/program-assessment/) recommended the inclusion of direct, indirect, and supportive evidence. Direct evidence, which can be the student artifacts from course work like exams, capstone projects, and portfolios, can provide information on what the students know and can do and as such implies the achievement of the learning outcomes. Perceptions of students, alumni, employers, and other stakeholders comprised the indirect evidence for inclusion in the program and course evaluation. One example cited in the University of Toronto material is asking alumni the extent to which the program that they had completed at the university prepared them for their current position. Note that answers to this question can provide information as to the continuing relevance of the program in addition to the perception of the alumni. The supportive evidence are aspects connected to student learning like graduation rates, job placements data, faculty-to-student ratio, and program promotional materials, among others.

The nine principles of good practice in ODDE by Stewart and Kogan (2015) can also provide some insights on what should be looked into in program and course evaluation. The nine principles are student-faculty contact, cooperation among students, active learning, prompt feedback, time on task, the communication of high expectations, respect for diverse talents and ways of learning, the establishment of (clear) course procedures, and the effective use of technology. Likewise, the evaluation questions forwarded by Burns (2018) can help in determining the data-gathering mechanisms which can ensure the technical soundness and rigor of the evaluation process. For instance, the “what” questions usually require quantitative designs while the “how” and “why” questions which usually look at the process imply qualitative designs. Mixed method designs can also be considered depending on the type of data or question being asked as part of the evaluation process.

While the foregoing recommendations may present a comprehensive perspective for course and program evaluation, one can also look at the traditional role of education in terms of its contribution to national and global concerns like sustainable development and inclusion in education as other relevant components. Specifically on inclusive education, the higher level of modern ICT integration in ODDE may put this system of education to a better position to provide inclusive learning opportunities targeting the inclusion of nontraditional learners who cannot be part of the conventional education system. It should also be noted that the essence of our basic guiding principle in the ODDE system is inclusion, hence, the extent by which this purpose is achieved should be a major component of the evaluation framework.

Innovating on the Program and Course Evaluation: The Role of Technology

The emerging trends in education which provide the scenario into the future of learning imply the need for an innovative approach to program and course evaluation especially in the ODDE system where such innovations may happen sooner compared to the conventional mode of instruction. These trends which include the digitization of textbooks, deployment of immersive technologies like the Virtual Reality into the teaching environment, more mobile learning practices (Gajura, 2020), and the move toward more digital learner-learner, learner-teacher, and learner-content interactions imply the need to consider the data that can be extracted from the virtual learning environment. Benigno and Trentin (2000) suggested the analysis of course messages in terms of number and content and the activity logs or records of the activities of the students which are automatically recorded in the virtual learning environment or Learning Management System. Other emerging trends, like microcredentialing which can be concretized through the offering of MOOCs, stackable credits or unbundling of the traditional degree programs, and AI-driven teaching and learning processes like e-tutoring and automated assessments, also require innovative perspectives on program and course evaluation vis-a-vis the needs and gaps that they are trying to address in the overall scheme of manpower and economic developments. Given that these trends are anchored on digital tools and platforms, a data analytics-driven process is a logical direction for program and course evaluation. It should be noted that the use of data analytics is already a common practice in business but is still at its infancy when it comes to higher education (Dziuban, Moskal, Cavanagh, & Watts, 2012). This potential, however, had already been recognized by software developers who are now offering digital solutions for a fully automated course evaluation process, analysis of data, and distribution of results (https://explorance.com/solutions/course-evaluations/).

Conclusion

The existing QA Frameworks for ODDE can provide a perspective on what should be considered when evaluating ODDE programs and courses. However, there are no standardized terms, parameters, and indicators as the different frameworks use different terms to refer to the same thing (e.g., program and curriculum) or use the same term (e.g., course) to refer to different aspects which can cover course design, course materials development, and course delivery. This may have resulted in the lack of a generally accepted understanding of quality when it comes to ODDE but, at the same time, points to the complexity of the evaluation process in this system of education. This further emphasizes the need for a standard program and course evaluation framework for ODDE as recognized and recommended by the various authors. The aspect of program openness and inclusion, which resulted in the diverse profile of the learners and which, in turn, impact on course design and delivery, was not given enough emphasis in the articulations of quality in the QA Frameworks studied.

While the existing QA Frameworks can serve as a very good starting point toward the development of the standardized program and course evaluation since they already provided an extensive articulation as to what should be the indicators of quality in this system of instruction, the emerging and projected trends in the higher education system like microcredentialing, stackable credits, the use of blockchain technology, and the likes will have an impact on the process, the data, and the indicators or parameters for program and course evaluation. Data which reflect the different teaching and learning processes and which are automatically captured in the virtual environments or digital platforms where such processes take place will definitely change the framework, and possibly the focus, of program and course evaluation in ODDE. This also implies the need to employ advanced research methodologies for ODDE institutions to be better prepared for new challenges emerged from the future learning environment.

Cross-References