Toward Process Variability Management in Online Examination Process in German Universities: A State of the Art

Abstract

In contemporary organizations, multiple variants of the same business process are often considerable. Such business process variability has caused considerable challenges, both while modeling processes and in their execution. In order to develop a new approach to managing process variants, or extend an existing one, in this research, we review the state of the art in a particular area: online examination processes. We show to what extent variability should be considered in exam processes, whether this is due to special legal restrictions and regulations, different exam frameworks, or even different technical infrastructure. This could be the foundation for developing an approach to managing process variability in the field of e-assessment. Initial findings indicate that examination processes have many similarities, but also considerable differentiation. Therefore, there an appropriate model needs to be developed in order to manage variability in e-assessment and the developed approach must then be validated in identifying faculties. This paper constitutes a first step in this direction.

1 Introduction

In today’s dynamic world, there are often multiple variations of identical business processes. Rosemann and colleagues noted, for instance, that SAP offers 27 different industry solutions with corresponding business process reference models (Rosemann and van der Aalst 2007). These models usually include decisions in the workflow, which could be made before executing process instances. It is impossible for both variants of such decisions to coexist in a certain domain or process context. However, conventional modeling approaches do not offer the opportunity to differentiate between such decisions and regular decisions during the runtime of a process instance. An important element of controlling the variability in business process models is to separate the usual runtime decisions from decisions at configuration time, called variation points. The results of such steps are complex artifacts. The number of artifacts makes the manageability of related workflows more complex. Based on the reviewed literature, organizations take different approaches to managing process variability (Ayora Esteras 2012). The existing approaches have limitations in terms of supporting an entire set of elements like control flow, rules, and legal regulations during the construction and execution of business processes.

For this paper, we chose an educational field as an example of process variability, in order to observe effects and causes of variability. The goal is to have a comprehensive overview to address the problem of process variability in online examination processes at German universities and show the necessity of managing it through an appropriate business process model. To achieve this goal, we set out the state of the art in research regarding the process variability of online examinations from different perspectives. This could be the basis for developing a process model to manage existing variability in this field. We evaluate existing approaches and concepts in the context of e-assessment in the literature and clarify current accreditation processes in educational fields. This will help to identify to what extent existing examination procedures reflect variability and demonstrate the necessity of developing a unified e-assessment model to cover all variability in the learning-teaching process.

The paper is organized as follows: After illustrating motivation through existing studies in the following section, the research method is explained. After exploring the literature and data collected in the identified domains, the results are evaluated. Finally, the need for further work is explained in the conclusion.

2 Motivation

In this section, we have a close look at process variability and its challenges, identifying the importance of variability management in the organizations. Basically, process models capture an organization’s activities in achieving certain business goals. The aim is to better understand the process, its implementation, and its execution in a workflow (Becker et al. 2013). However, there are a lot of possible variants for one process. Such business process variability creates considerable challenges in process modeling and execution:

• the variants may be modeled in a highly redundant way, so there are many identical or similar parts;

• there is no strong relation between the variants, so there is no support for automatically combining existing variants;

• manual modeling in every single process variant would be time-consuming and error-prone.

In recent years, the proper management of business process variability has been the subject of numerous scientific studies. A very comprehensive survey article about business process variability can be found in Valenca et al. (2013), which describes more than 80 primary sources. Based on this study, significant numbers of variability approaches exist, where each one addresses different issues in terms of process variability. Valenca and colleagues observed 57 new approaches to different aspects of variability in processes (Valenca et al. 2013). They divided these references into five categories: business process configuration, to capture an instance of the reference model; business process correctness, to semantically support correction of the process model; business process flexibility, to change process models fast and easily; business process modeling, to visualize variability in process models; business process similarities, to investigate differentiations between business process models. It is argued that only 30% of solutions are practically evaluated through case studies and surveys, especially of industry: The lack of empirical studies in process variability is considerable, with implications for executing process variability (Valenca et al. 2013).

In the public sector, as in business, processes have a lot in common, but significant differences due to the local conditions and legal regulations are considerable. Vogelaar and colleagues analyze and compare the different processes of ten Dutch municipalities which are found to vary in terms of classical standardization processes (Vogelaar et al. 2012).

In education, Arnold and Laue studied controllability of variability in examination process models (Arnold and Laue 2014). They investigated six different courses at three different universities in the German Federal State of Saxony, to achieve better comparability. Based on this research, considerable variability in examination processes exists, even in one university between different fields of study. The authors tried to provide a solution based on existing variability approaches, in order to manage examination processes. They argued that appropriate process variability modeling requires modeling skills and significant experience in the identified domain (Arnold and Laue 2014).

We focus on online examination processes in higher education, presenting the state of the art in three different domains and observing existing process variability in order to gain a comprehensive overview of process variability in this field, highlighting the need for this to be managed.

3 Research Method

The purpose is to evaluate the existing variability in higher educational e-assessment processes, as a basis for further research into variability management in this field.

The state of the art is identified in five phases (Cooper 1998):

• 1. Problem formulation: The academic goal and research relevance are defined;

• 2. Literature search and data collection: Literature and data related to the problem formulation are identified;

• 3. Literature evaluation: The acquired literature is assessed for relevance and categorized;

• 4. Interpretation: The results are analyzed and interpreted;

• 5. Presentation: The results are presented in a suitable fashion.

Assessments are an important part of the educational cycle (Ferrão 2010) and have a great impact on the learning process. They provide valuable information about the effectiveness of a study course in increasing the students’ knowledge (Primiano et al. 2004). An appropriate assessment process is not only important in terms of teaching and learning, but also for accreditation processes and educational standards (Ferrão 2010). Recent developments in e-learning can be seen as an accelerator to developing e-assessment alternatives. It is therefore becoming more important to develop methods for e-assessment and to gain feedback on learning and teaching (Sangi and Malik 2007). Furthermore, Dermo has shown that e-assessment can offer different forms of assessment with immediate feedback to both students and lecturers, so it can be recognized as a complementary tool in the learning framework (Dermo 2009). Exam regulation documents, which are the basis for accreditation processes in higher educational institutions, have a lot in common. But in some points, they differ from one university to another or even from one course to another within the same institution. Therefore, there is variability in assessment processes, which is an obstacle to developing a unified process model for e-assessment.

• We reviewed the state of the art in three identified domains within e-assessment:

• IT-related approaches;

• designing study courses and e-assessment concepts;

• accreditation process.

In the following, the results of our literature review and data collection in each domain are explained separately.

3.1 Literature Search and Data Collection in Three Domains

3.1.1 Domain: IT Approaches

• Data sources: AISel and EBSCO

• Research period: 2000–2013

• Search terms and keywords: e-assessment, education, online examination, e-test, computer-based exam (in abstract and title)

• Number of related articles:

  • AISel: 51 after reviewing and removing doublets and non-related articles: 14

  • AISel: 51 after reviewing and removing doublets and non-related articles: 14

EBSCO: 62 after reviewing and removing doublets and non-related articles: 18

Based on the reviewed articles, a main classification can be recognized in the context of e-assessment in higher education:

• pedagogical issues (educational view)

• technical issues

Different kinds of terms and concepts are used, based on pedagogical and technical approaches: Each one addresses one or more aspects of e-assessment. These issues are summarized in the following Tables 1 and 2.¹

Table 1 Pedagogical aspects of e-assessment

Table 2 Technical aspects of e-assessment

Most of these references include multiple issues from the technical and pedagogical perspectives. These issues are connected and cannot be separated.

Jacob and colleagues deploy an e-assessment tool, the Black Board Learning System (BBLS), as a comprehensive e-learning software to facilitate continuous assessment and evaluate its effects on learning processes (Jacob et al. 2006). It reveals that the biggest advantage of e-assessment in this system is immediate feedback, which bolsters the formative assessment.² One weakness of the system is the lack of the automatic evaluation of essay-writing exams. Kehily analyzed the impact of a Web-based e-learning platform that can support effective teaching (a course management system for lecturers) and formative assessment (a computer-assisted learning tool for students) in a case study (Kehily 2011). Venkatraman developed a four-step student-centered approach to an effective e-learning process and, in a case study of information system (IS) courses, evaluated this approach for different assessment methods including individual, group, peer, and self-assessment (Venkatraman 2007). These four steps are:

• 1. understanding the students’ learning style and their learning expectations;

• 2. identifying suitable assessment models;

• 3. designing a set of assessments;

• 4. evaluating the impact of the assessment on the learning process.

Dermo evaluates the possible risks in planning e-assessments such as computer stress, fairness of choosing questions randomly from a bank, accessibility, and the contribution of e-assessment to students’ learning, through six dimensions in a case study (Dermo 2009). These six dimensions are: affective factors, reliability and fairness, validity, security, practical issues, and teaching and learning terms, which are a mixture of pedagogical and technical issues. Daly and colleagues argue that existing e-assessment solutions focus on developing technical and infrastructural issues more than educational aspects (Daly et al. 2010). McCann identifies different factors which affect real implementations of e-assessment systems based on two IS theories: Roger’s theory³ and Eckel and Kezar’s theory⁴ (McCann 2010).

3.1.2 Domain: Designing Study Courses and E-Assessment Concepts

• Data sources: German university homepages

• Research period: 2000–2013

• Search terms and keywords: e-assessment, online examination, project, computer-based exam, e-exam, e-test

In this domain, we began by finding some case studies of e-assessment or online examination at German universities. In order to have an appropriate sample, we selected universities which conducted online examinations or had a project to recognize a unified approach or process regarding e-assessment in Germany. Different kinds of projects in terms of computer-based examinations have been in progress from the year 2000 onward. Table 3 summarizes all these projects with their functionality and their relation with exam regulation documents.

Table 3 E-assessment projects in German universities

Of all the universities studied, only the University of Duisburg-Essen proposes a process model for implementing online examination. Proposing such a process model for online exams has the following advantages:

• improves the speed of feedback to students;

• motivates students and lecturers to increase computer skills through the integration of multimedia elements such as audio and video, and using complex digital systems;

• significant time saving for lecturers by automatic correcting process.

By reviewing the exam regulations and conditions, it becomes obvious that there is no identified exam process model in the administrative processes at different universities. A process model not only supports understanding the complexities of processes properly, but also contributes to advancing and improving defined processes (Irani et al. 2000). Therefore, in order to understand the examination workflows carried in the universities, a process model is required to analyze exam processes comprehensively.

3.1.3 Domain: Accreditation Process

• Data sources: Accreditation agencies authorized by AISel⁵ and EBSCO

• Research period: 2000–2013

• Search terms and keywords: e-assessment, education, online examination, computer-based exam, accreditation process, accreditation criteria (in abstract and title)

As a definition, accreditation is a criteria-based procedure to assess and evaluate the admissibility of an educational program in terms of quality (Gorgone 2006; Impagliazzo and Gorgone 2002; Reichgelt and Yaverbaum 2007). The main goal of accreditation is to assess the educational quality of an academic program to ensure that it meets certain quality standards, called accreditation criteria (Reichgelt 2007).

Based on the European Network for Quality (ENQA), each educational program should fulfill the minimum in the following set of requirements to be accredited:

• requirements and objectives;

• teaching–learning process;

• learning resources;

• monitoring, analysis, and overview.

The assessment or examinations are placed in the teaching–learning process, which is the most complex aspect of this model because it includes a mixture of technical, pedagogical, and social competences and, furthermore, there is a great freedom to manage courses in order to achieve identified objectives.

According to Reichgelt and colleagues, there are two accreditation types:

• institutional accreditation is applied to an academic institution, like an entire university;

• specialized accreditation is applied to a subunit in an institution and includes two levels: program accreditation focuses on the content of the program, while administrative accreditation concentrates on the administrative process in a subunit of an institution (Reichgelt and Yaverbaum 2007).

The authors explain that there are two main approaches to accreditation processes. The first one is the input-based approach which measures various minimal standards through a checklist based on the learning-input processes such as curriculum, teaching resources, library, laboratory, and other facilities. The second is the outcomes-based approach, which considers the program’s outcomes, such as the institution’s educational objectives and student learning. Reichgelt and colleagues argue that a significant shift from the input-based to the outcomes-based approach has occurred in recent years and academic institutions attempt to conform themselves with outcome criteria (Reichgelt and Yaverbaum 2007).

3.2 Accreditation Processes in Germany

In Germany, the federal states are responsible for accreditation processes and at least 11 authorized accreditation agencies in different fields of education (medical, natural science, engineering, economics, etc.) are in operation at present.⁶ Educational accreditation in Germany is based on two issues:

• 1. standards of study courses and degrees, which are based on regulations for study and examinations;

• 2. accreditation in teaching, which is based on quality improvement measures.

The requirements in the examination accreditation process are as follows:

• examinations are coordinated so that students have sufficient time to prepare themselves;

• it must be possible to move directly from the bachelor’s degree to the master’s degree without loss of time;

• the form of examination is laid down in the description of each module;

• examinations should not cause extensions to the period of study;

• the evaluation criteria are transparent for both lecturers and students;

• the degree program ends with a final thesis;

• it is checked whether students are capable of oral discussion in their specialist area;

• the supervision of the final thesis is subject to precise regulations in the curriculum.

The archived documents in the accreditation process are test results, drop-out rates, any quantity results of examinations, as well as feedbacks from the courses.

This survey of three domains indicates that multiple perspectives exist, which cause variability in performing and evaluating e-assessment processes. It is therefore essential to develop an appropriate model to account for this variability in online examination processes.

4 Literature and Results

This literature review was performed in order to demonstrate existing process variability in e-assessment in different domains of higher education, which occurs for different reasons. The results of each domain and its relations to process variability are analyzed and presented in the following.

4.1 Evaluation of IT Approaches

Based on reviewed IT approaches, it can be argued that due to the great advantages of and positive impacts of online exams on learning processes, there is a considerable movement from traditional to electronic assessment in academia today. Furthermore, studies⁷ show that different issues, from pedagogical to technical and even social matters, cause process variability in higher education examinations. To identify a way of managing this inherent variability and to construct unified e-assessment procedures, it seems necessary to have a cross-functional view of e-assessment projects. Some of the issues related to e-assessment are: the impacts on student learning; effects on the teaching method; formative and collaborative e-assessment; immediate feedback and legal issues; evaluating possible risks in planning such as computer stress or fairness impression; developing infrastructural issues, such as security and question banks.

4.2 Evaluation of Study Courses

Thirty German universities from different federal states and different study courses were reviewed.⁸ The results show that e-assessment or online examination projects are currently in progress in 18 universities, but just nine are performing such assessments in practice. It should be noted that although online examinations are in use at some universities, there exists considerable process variability, too, which makes it difficult to extract a unified platform in this area.

Furthermore, legal conditions in regulation documents create practical limitations on performing online examinations. In order to make it an acceptable form of assessment, electronic examination should be referred to in a paragraph or even a sentence in the corresponding exam regulation document. Based on reviewing examination rules documents in all these universities, Table 4 reveals that only three German universities have a paragraph stating that online examination is an admissible examination form.

Table 4 Summary of e-assessment projects in German universities

4.3 Evaluation of Accreditation

The results revealed that accredited educational programs are subject to a variety of quality criteria. These criteria depend on the level and the goals of the courses. Based on the Information Model from the European Network for Quality (ENQA), assessment and examination processes are placed within the teaching–learning process, which is one of the complex parts of accreditation. It involves different aspects such as pedagogical, technical, legal, and even social issues.

No framework for examination forms (traditional or computer-based) was identified in this study of the German accreditation process. In exam regulation documents, assessments were only described in the terms listed at the end of Sect. 3.3 above. The form of examinations is not restricted by the accreditation process, but is under the authority of the educational systems and based on the identified objectives of courses. This causes process variability from one educational institution to another.

4.4 Summary of the Results

For clarity, we summarize the results of the three domains in Table 5.

Table 5 Summary of the results

In sum, exam processes show variability for the following reasons:

• variability in legal restrictions and regulations in different educational institutions;

• variability in exam framework, not only in different universities but also in one university between different courses;

• variability in study course design;

• variability in technical infrastructure for online examination processes.

Based on the obtained results, it can be argued that an appropriate design for variability management has to be aligned with the specified domain of projects and conditions to cover all variability within the identified domain. There is only one process model for performing online examinations in some study courses at University of Duisburg-Essen, which could be an appropriate model for conducting online examinations at German universities.

We studied existing process variability (effects and causes) in online examination processes from different perspectives. We found that to manage process variability, an appropriate business process model is necessary to cover all examination processes and features of educational institutions.

According to the existing literature, one approach to process variability management is the single model approach, which models all known variants of the process in one common model (Hallerbach et al. 2010; Kumar and Yao 2012). The alternative is to model every variant of a single model, which is called the multi-model approach. The latter models will have a simpler structure (Hallerbach et al. 2010; Kumar and Yao 2012). Some advanced modeling approaches explicitly deal with families of process models, such as configurable event-driven process chains (C-EPCs) (Rosemann and van der Aalst 2007), PROcess Variants by OPtions (PROVOP) (Hallerbach et al. 2010), ConDec (Pesic and Aalst 2006), and feature modeling, which is normally used in software engineering. Each of these approaches has their own advantages and drawbacks but there is a lack of adaptability between different approaches (Valenca et al. 2013).

The next step in this research is to conduct a comprehensive overview of available variability approaches, to identify which of them are more appropriate for online examinations in higher education. Further research is needed to evaluate how existing process variability approaches can be compatible with exam processes, and to what extent existing process variability in this field can be controlled and managed.

5 Conclusion and Further Work

The aim of this research is to demonstrate existing variability in examination processes and emphasize the need for variability management. This could be the basis for developing a new approach to business process variability, or extending an existing one. To reach this goal, we concentrated on online examination processes in higher education.

As a preliminary stage, we performed a literature review in three different domains: IT approaches; concepts; course design and the accreditation process. This paper demonstrates the important role of business process management in improving and promoting the design of e-assessment processes.

Combined with this literature review, an analysis of current e-assessment case studies and projects in German universities yielded the following results. Although different kinds of projects under the name of e-assessment or online examination are in progress in German universities, the variability in these processes is recognizable. In other words, similar processes exist, but distinctions and variability are observable as well. Furthermore, the review of the exam regulation documents for various study courses at different universities revealed that these regulations do not yet mention an acceptable framework for performing e-assessments.

Process variability exists in e-assessment at German universities. It is necessary to manage this variability through an appropriate business process model to support online examination procedures.

This paper describes research in progress which clarifies and identifies the necessity of developing an existing approach to manage and control the process variability in university e-assessment. The next step is to study process variability management to identify an appropriate model in the context of e-assessment. This could be followed by the development of a prototype for the identified approach and finally the validation of the developed method.

Appendices

Appendix 1: Summary of IT Approaches to e-Assessment

No.	Reference (author, year)	Summary
1	F. Braun, 1998	It introduces some web-based tools for online assessment such as Learning space, Web forms, Microtest, and Quizmaker.
2	Charoen, 2007	It indicates the benefits and limitations of implementing an e-learning system in an educational institution in Thailand based on expert interviews.
3	Dittman, 2008	It introduces a structure, the collaborative e-learning template (CET), to improve collaborative activities in the learning process such as collaborative assessments through a course.
4	Papp, 2000	It indicates different critical factors in distance learning.
5	Squires et al., 2004	It is about education policy analysis in Arizona University.
6	Shen et al., 2000	It analyzes the effectiveness of an online collaborative examina-tion process through a questionnaire.
7	D. Hall, 2010	It introduces some tools to assess teachers.
8	Jacob et al., 2006	It reveals the importance of assessments in the learning process through different learning approaches. The Black Board Learning System (BBLS) is used as a comprehensive e-learning software platform in the context of continuous assessments.
9	Campbell, 2008	It proposes an application called the Electronic Performance Support System (EPSS) to enhance the performance of assess-ment tasks.
10	Kehily, 2011	It uses a web-based platform, DIT learning teaching and technol-ogy center, as a course management system for lecturers and as a computer-assisted learning tool for students. It is a case study in the Dublin Institute.
11	Attali and Burstein, 2006	It proposes a flexible modeling procedure which can be used to as a basis for expert judgment.
12	Venkatraman, 2007	It is a case study which develops a four-step student-centered approach to increase positive impacts on the learning process. It analyzes the strengths and weaknesses of assessments in IS courses through a survey.
13	Maheswari, 1998	It is an empirical study of technical problems in web-based sys-tems for assisting education.
14	Sybol, 2005	It investigates how use of an online assessment tool, the Com-puter-Assisted Personalized Approach (CAPA), helps achieve teacher and students learning goals. It is a case study of urban courses in Florida University.
15	Webb, 2010	It evaluates how digital technologies such as tablet PCs, wireless technology, and Web 2.0, can facilitate formative and collabora-tive assessment effectively.
16	e-learning age magazine, 2011	It is a case study of The Association of Chartered Certified Ac-countants (ACCA), which wants to deliver all accounting examina-tions electronically through an e-assessment program.
17	Miller, 2011	It is a case study of the positive role of aesthetics design on learner perceptions and task performance in an e-assessment environment.
18	Dascalu and Bodea, 2010	It proposes a qualitative analysis of an e-assessment and its role to increase knowledge creation and knowledge management.
19	Sainburg and Benton, 2010	It is a case study of the importance of formative assessment in the learning process in schools.
20	Dermo, 2009	It evaluates the possible risks in planning e-assessment through six identified dimensions in a case study.
21	Jordan and Mitchell, 2009	It explains some methods to facilitate answer and assessment of free-text questions in e-assessment. It examines its model (IAT) in the UK Open University, using information extraction tech-niques and NLP.
22	Ferrao, 2010	It shows that an e-assessment system can be a good alternative to encourage students and has positive effects on the learning process.
23	British Journal of educational studies, 2009	It summarizes different works which all encourage educational institutes to use e-assessment technologies to support the learn-ing process, with emphasis on feedback from e-assessments.
24	Hodgson and Y. C. Pang, 2012	It is a study of statistics students to evaluate the advantage of formative assessment.
25	Glenberg, 2010	It examines the impact of formative quizzes on knowledge con-struction.
26	Daly et al., 2010	It argues that existing assessment solutions focus on developing technical and infrastructure issues more than educational as-pects of assessment through different educational ases.
27	McCann, 2010	It is a case study of a US campus to explore how a new e-assessment system is implemented and to identify factors which affect how it is adopted.
28	Nicol, 2007	It suggests a set of principles for the effective design and evalua-tion of a formative assessment and feedback process through two different case studies.
29	Boyle, 2010	It is about forecasting models like the Bass model, used to evalu-ate how educational systems can adopt themselves to e-assessment methods.
30	Boyle and Hutchison, 2009	It makes the case for the substantial effects of e-assessment on the assessing process and the technical difficulties involved in developing sophisticated assessment tasks.
31	McNeil et al., 2011	It looks at e-assessment from both sides: educational and tech-nical. It develops a model for analyzing the life cycle of institu-tional assessments.
32	Sangi and Malik, 2007	It reviews existing e-assessment models and practices and identi-fies the challenges facing developers in South Asia.

Appendix 2: List of German universities reviewed

• 1. Freie Universität Berlin

• 2. Universität Bremen

• 3. Universität Duisburg-Essen

• 4. Justus-Liebig-Universität Gießen

• 5. Philipps-Universität Marburg

• 6. Johannes Gutenberg-Universität Mainz

• 7. Karlsruher Institut für Technologie

• 8. Rheinisch-Westfälische Technische Hochschule Aachen

• 9. Hochschule Koblenz

• 10. Leibniz Universität Hannover

• 11. Westfälische Wilhelms-Universität Münster

• 12. Universität Koblenz-Landau

• 13. Universität Regensburg

• 14. Universität Kassel

• 15. Technische Hochschule Ostwestfalen-Lippe

• 16. Technische Hochschule Wildau

• 17. Hochschule Fulda

• 18. Universität Trier

• 19. Universität zu Köln

• 20. Albert-Ludwigs-Universität Freiburg

• 21. Universität Düsseldorf

• 22. Hochschule Wismar

• 23. Universität Ulm

• 24. Eberhard Karls Universität Tübingen

• 25. Georg-August-Universität Göttingen

• 26. Rheinische Friedrich-Wilhelms-Universität Bonn

• 27. Otto-von-Guericke-Universität Magdeburg

• 28. Technische Universität Dresden

• 29. Universität Leipzig

• 30. Technische Universität Chemnitz