Keywords

1 Introduction

This paper is divided into 7 parts and starts with Introduction laying out the context and background about Virtual Mobility (VM) in Europe as well as the aims and outcomes of the Open Virtual Mobility project. The next Sect. 2 introduces design approaches applied in developing the VM Learning Hub showing how the demands in European Higher Education Area (EHEA) and Open Education (OE) principles shape the design considerations at various levels of user experience design: (i) system design, (ii) interaction design, (iii) content design, (iv) learning design and (v) visual design, following the UX model by Garret [1]. The concept and the design approach of the VM Learning Hub are presented in Sect. 3. Section 4 is dedicated to the VM Learning Hub components, taking into consideration the five dimensions of UX by Garret [1]. Section 5 presents preliminary research results from the VM concept mapping study with VM experts from Europe and discussed the implications for the next design iterations of the VM Learning Hub. Finally, Sect. 6 lays out conclusions addressing specific challenges and limitations in designing collaborative learning experiences in context of Virtual Mobility. The following sections will focus on five central aspects of the collaborative learning design:

  1. 1.

    Open Education approach to promote the achievement, assessment and credentialing of VM skills (including open and flexible approaches to designing transnational online collaboration) and the UX model - both building a framework for the development of the VM Learning Hub as laid out in this paper.

  2. 2.

    VM Learning Hub as a Collaborative and Personal Learning Environment with learner-centered forms of skill recognition (e.g. open digital credentials), collaboration (e.g. algorithm-based matching tool based on psychometrics) and access (e.g. mobile app).

  3. 3.

    Enhancing learner-experience by means of meaningful gamification design, e.g. customization of goals, tracking of team performance and visual design for the enhancement of learning experience.

  4. 4.

    Competency-based forms of e-assessment such as assessing VM skills and experiences from diverse learning contexts, a quality-assured and user-friendly tool to pre-assess own VM skills and recommendations for resources to close skill gaps.

  5. 5.

    Semantic description of VM skills using machine-readable competency vocabulary and cross-referencing skills in a relational graph to allow identification of similarity between VM skills as a basis for e-assessment and recognition of VM skill sets (e.g. digital, intercultural, language, collaboration skills).

1.1 Virtual Mobility in European Higher Education Area

The European Higher Education Area (EHEA) is framework based on public international law standards such as Lisbon Recognition Convention and part of the Bologna Reform aiming at ensuring more comparable, compatible and coherent systems of higher education in Europe. The European Higher Education Area has been in operation since March 2010 and during the last eighteen years 48 countries have been continuously building this common education area by adapting their higher education systems through the implementation of reforms in higher education on the basis of common values, facilitation of fair recognition of foreign qualifications and/or study periods abroad and enhancement of mobility of students and staff. Some of the key instruments implemented in EHEA include the European Credit Transfer and Accumulation System (ECTS), the Diploma Supplement and Learning Agreements for Studies ensuring the recognition for the activities successfully completed abroad. These tools and processes have contributed to a higher compatibility, internationalisation and quality assurance in higher education in EHEA in the recent years.

Mobility of students and staff has been one of the central objectives and main policy areas of the EHEA. For example, the Communiqué of European Ministers for Higher Education in Europe from 2009 states that: “In 2020, at least 20% of those graduating in the European Higher Education Area should have had a study or training period abroad” [2]. Mobility has been considered an important part of higher education as it supports personal development and employability, fosters respect for diversity, encourages linguistic pluralism underpinning the multilingual tradition of Europe and increases cooperation and competition between higher education institutions [2]. The Erasmus program, superseded by Erasmus+, has been one of the most well-known programs promoting mobility of students and staff.

However, as the mobility statistics show, despite numerous initiatives and programs, the uptake of mobility of students and staff has been very diverse across Europe [3]. Despite acknowledging the social and cultural benefits of mobility for higher education, awareness and exploitation of mobility instruments are still not as extensive as anticipated [4]. A study about the obstacles to student mobility lists a number of reasons preventing students to be mobile [5]. These include socio-cultural background and status, disabilities and chronic diseases, family and parental obligations, the financing of the mobility period, the language proficiency, the availability of information about the mobility period and the recognition of study periods and degrees [5]. From this perspective, the concept of virtual mobility has been discussed as a non-discriminatory alternative of mobility. Through the use of ICT supported activities organized at the institutional level mobility becomes accessible to all [6]. It is important to emphasise that virtual mobility is “organized at the institutional level” which means that the activities are not ad hoc initiatives by single teachers or individual students, but are fully embedded in mainstream and core processes of the institution [6]. To conclude: Virtual mobility has a great potential to contribute to the internationalisation, innovation and inclusion in higher education, and the main obstacles to mobility can be dramatically reduced by adding the virtual component.

1.2 The Project Open Virtual Mobility

The project Open Virtual Mobility (openVM) is a strategic partnership for innovation and the exchange of good practices. The partnership is composed of nine European partner organisations from higher education, aiming at enhancing the uptake of virtual mobility in higher education by improving VM skills and in consequence VM readiness. The possibilities of virtual mobility (e.g. joint virtual seminars, virtual internships or placements, virtual campuses or virtual support activities for physical mobility before, during and after physical mobility) are still unknown to many educators and students in Europe [4]. The study by Dauksiene et al. [4] pinpoints some of the key barriers impeding VM implementation on a wider scale in higher education in Europe. These include a lack of knowledge and/or the lack of consensus about the concept of VM on the national and institutional levels, missing practical examples and applicable scenarios for VM implementations, lack of knowledge and/or experience in recognising and accrediting VM activities, and missing evidence about the effectiveness of VM at different levels (e.g. contribution to physical mobility, intercultural experience, internationalisation).

The project Open Virtual Mobility addresses these challenges and aims to create the European Virtual Mobility Learning Hub for achievement, assessment and recognition of VM skills. The VM Learning Hub is envisaged to become a central reference point for educators and students wishing to learn about the different possibilities and forms of virtual mobility, collaborate on designing VM activities, assess and recognise VM skills, i.e. skills acquire from and/or relevant for the implementation and/or participation in VM actions. The project aims to develop a set of tools, methods and guidelines to enhance achievement, assessment and recognition of skills, provide support on pedagogy and technology for the design and implementation of virtual mobility, and enhance collaborations of participating organisations, educators and students. The VM Learning Hub aims to provide engaging and effective learning experience and to provide evidence about how achievement, assessment and credentialing of VM skills contributes to the uptake virtual mobility.

The premise of the openVM project is that virtual mobility can develop its potential, provided higher education leaders, educators, students and other relevant stakeholders, such as International Offices, know about and know how to use the opportunities of virtual mobility. This means higher education staff, educators and students need the necessary skills, confidence and readiness to initiate and implement VM activities in their own organisations. The openVM project intends to enhance the readiness for virtual mobility against the backdrop with Open Education (OE) and addresses the need of creating accessible opportunities for achievement of skills, needed to design, implement and participate in VM activities in line with the principles of Open Education.

2 Overall Design Approach in the Project Open Virtual Mobility

The design approach in the project Open Virtual Mobility is based on the concept of Open Education (hence open in the name of the project). Open Education is an umbrella term under with many different understandings. In Europe, particularly in higher education, Open Education has been discussed as an important element of the European policy agenda. The key perspectives on open, higher education in Europe include (a) reducing or removing access barriers such as financial, geographical, time and entry requirements barriers, (b) modernising higher education in Europe by means of digital technologies, and (c) bridging non-formal and formal education, by making it easier to recognise learning achievements [7]. Both virtual mobility and open education aim to enhance participation in international knowledge flows, use of digital media, improve teaching and learning, attract and keep talents through internationalisation, but also innovate and build capacity.

The project Open Virtual Mobility focuses on creating readiness of educators and students for virtual mobility against the backdrop of open education. The model applied in the project is the OpenEdu framework developed by the Joint Research Centre of the European Commission. The OpenEdu framework is based on the results from four studies on OE in Europe, which included desk research, reviews of academic and grey literature (websites, blogs, newspapers, reports), consultations and validation with experts and final validation by the target audience, i.e. the decision makers at universities across Europe [7]. The framework builds on the information on the state of-the-art of OE in Europe and provides a holistic view of OE in higher education. The framework proposes 10 dimensions of OE, i.e. 6 core dimensions: access dimension, content dimension, pedagogy dimension, recognition dimension, collaboration dimension and research dimension, and 4 transversal dimensions: strategy dimension, technology dimension, quality dimension and leadership dimension. The core dimensions represent what is included and transversal dimensions indicate how to achieve it [7]. Each dimension interrelates with all other dimensions and allows for different degrees of openness in higher education. The model recommends to apply a holistic strategy for opening up higher education along these 10 dimensions and to involve various stakeholders in the process of design, i.e. education providers (institutions), teachers, researchers, students and policy makers. The design approach in the openVM project is based on the recommendations of the OpenEdu framework and applies the proposed descriptors as listed in Table 1.

Table 1. Dimensions of Open Education proposed by the OpenEdu framework [7].
Full size table

The design of the collaborative VM Learning Hub in the project Open Virtual Mobility applies the principles of Open Education (OE) to promote achievement, assessment and recognition of VM skills. The VM Learning Hub aims to create online, open and flexible opportunities for higher education staff and students to learn about and how to plan and engage in VM, assess and recognize their VM skills and in this way increase readiness and confidence to implement virtual mobility.

3 Virtual Mobility Learning Hub Design Approach and Concept

The VM Learning Hub developed in the project Open Virtual Mobility is intended to become as a central reference point for achievement, assessment and recognition of virtual mobility skills. The components of the collaborative VM Learning Hub include an algorithm-based matching tool to enhance collaboration, an e-assessment tool based on rich evidence of learning from different contexts to assess VM skills, and open digital credentials to recognise VM skills with such as tools as Open Badges and Blockcerts. The VM Learning Hub is designed to cater for (a) collaborative activities as part of co-design of Open Educational Resources (OER) dedicated to virtual mobility (e.g. good practice examples and guidelines), and (b) collaborative activities as part of learning in the VM MOOC for achievement of VM skills as well as assessment and validation of VM skills (e.g. peer-reviews of evidence).

3.1 Design Approach

The approach to designing the VM Learning Hub builds on the principles of User Experience (UX) and User Engagement (UE). User engagement may be defined as the quality of the user experience as it emphasises the positive aspects of the interaction, such as being captivated and motivated to use technologies [8]. Designing for user engagement for collaborative learning means designing engaging experiences for groups of learners. User engagement has been associated with specific user characteristics which can be also applied to the field of learning, e.g. focused attention, sense of control, novel and unexpected experience, positive emotions during the interaction, willingness to repeat the experience [9]. User engagement in technology-enhanced learning setting can be evaluated by applying engagement metrics such as time spent on site, number of performed learning activities, number of comments [10].

The design of the VM Learning Hub in the project Open Virtual Mobility is based on the multi-layer user engagement framework proposed by Garrett [1]. The UX model by [1] is a user-centered design approach which defines the key considerations of UX design. This model has been already adapted to and applied in the field of technology-enhanced learning [10]. The design of user engagement in the VM Learning Hub takes place on all five layers as the elements affect each other to form the overall user experience, as shown in Fig. 1. Additionally, the VM Learning Hub is a suitable scenario to combine approaches for a modern web-application (task-oriented design) and a multi-medial hypertext-system (information-oriented design).

Fig. 1.
figure 1

UX elements in vertical layers including the horizontal duality of information-oriented vs. task-oriented design [1].

Full size image

3.2 Concept

From a user perspective the VM Learning Hub is a responsive web-application with a landing page that offers solutions for the different user needs mentioned before. Technically, several components will be loosely coupled to provide the functionality, but on the front-end users primarily interact with the learning hub interface. It will be based on the Open Source Learning Management System (LMS) MoodleFootnote 1, inspired by its MOOC-oriented extension Moodle AcademyFootnote 2.

As a start, users will have the choice of general information and first-step guidelines about conducting virtual mobility at their own organization (targeting teachers and administration) or about finding and participating in existing VM opportunities (targeting students). Alternatively, users can directly open e-assessment resources to assess their level of expertise and experience. In case these assessments do not require upload of artifacts as evidence, the result can be evaluated automatically. Finally, different entry points to online learning courses, depending on the users information need and level of expertise, are offered. On enrolment to a course users can voluntarily fill out the group formation questionnaires to be added to one suitable learning group. If they do not provide such personal data, they are matched randomly. Their peer learning group is created for assisting in task solving, actively responding to user forum questions, and the group is responsible for peer assessment and feedback which may be required to issue Open Credentials certifying VM skills.

All relevant activities, results and progress will be fed to an Experience API (xAPI) [11] via existing Moodle plugins [12] (or extensions to be written). The corresponding digital certification with the help of Open Credentials, such as Open Badges and Blockcerts will be provided by the partner badging platform as described in the sections below. The badging platform will watch the xAPI records and as soon as the user fulfills all criteria of a defined Open Badge, the user will be notified and the badge is issued. Official issuing organization is the Open Virtual Mobility consortium. Endorsements by national educational institutions with expertise in mobility in the EAHE are intended. To provide a consistent user experience, plugins for Moodle will display the achieved Open Credentials by connecting to the Representational State Transfer Application Programming Interface (REST-API) that provides this data. The design of the user interface and REST-API will be strongly inspired by Mozilla’s Open Source Backpack for Open Badges.

Interactive learning resources for individual and collaborative learning are designed and technically based on HTML5 Package format and supported by compatible tools such as the Open Source software H5P which enables users to create interactive content in an easy way [13]. Moodle plugins for adding H5P content existFootnote 3. Existing OERs will be transformed to, or extended by, the HTML5-based technology such as H5P. All content will be hosted in the Moodle platform. References to the individual resources, including metadata about VM skills will be stored in the badging platform in order to calculate most suitable learning pathways for users based on their xAPI records. To inform Moodle about these pathways and display them to users within the learning hub, a plugin will access the badging platform REST-API for these suggestions. The major benefit of loosely coupling the Moodle learning hub with the badging platform via xAPI and REST-API is the possibility for users to easily access, display, and export their Open Credentials. Likewise, users can access recommendations for learning resources and further Open Credentials beyond the context of virtual mobility.

4 Virtual Mobility Learning Hub Components

4.1 Learning Activities and Gamification

The VM Learning Hub aims to create engaging and effective learner experience through diverse, collaborative learning activities making use of gamification as an approach to enhancing collaborative learning. Gamification means using game elements in non-game contexts [14]. The design approach builds on the concept of meaningful gamification, which focuses on providing effective incentives by helping learners find meaning in each underlying activity [15, 16]. This approach is different to some of the traditional gamification approaches which tend to provide meaningless awards which may distract or even discourage learners. The concept of meaningful gamification goes beyond providing external rewards for reaching specified thresholds or levels, which can reduce the internal motivation of the learner [15]. The goal of meaningful gamification is to enhance the internal motivation without emphasising external rewards including popular scoring systems. Studies, such as the meta-analysis by [17] show that most forms of external rewards tend to reduce internal motivation [15]. Some of the other challenges of scoring-based gamification approaches include (a) limited possibilities for an individual to make choices without external influence or control (which may have negative effects on self-determination and self-regulation), (b) the necessity to keep the user in the reward loop all the time (which may have negative effects on learner autonomy as it never frees the learner from the external control of the scoring system), and (c) enhancing negative effects through the lack of progress or weak scores compared to other users [15]. Thus the approach of meaningful gamification applied in the design of the VM Learning Hub aims to avoid the pitfalls of score-based gamification approaches and focuses on allowing users to self-identify with the goals of activities by facilitating the understanding of the importance of an activity and helping users integrate the goals of activities with personal goals [15]. The key approach here is not to create goals for learning activities without user involvement, but to involve the user in the definition and/or customization of the goals so that the goals of activities can become relevant to user background, interests and needs [15]. This also includes taking into consideration the organisational context into of learning activities [15].

The meaningful gamification design of learning activities in the VM Learning Hub is applied to create a meaningful user/learner experience in view of enhancing readiness for the uptake of virtual mobility in higher education. The meaningful gamification design is applied to key learning and collaboration activities in the VM Learning Hub, such as (a) co-design of Open Educational Resources (OER), (b) collaborative learning activities in the VM MOOC, (c) peer-assessment of evidence as part of e-assessment of virtual mobility skills. Involving learners in co-designing OER and parts of the MOOC will be accomplished by using such approaches as “Crowd Creation” [18] and “Open Learning through Design” [19]. The co-design approach aims at creating learning activities and learning resources for educators and students wishing to develop their virtual mobility skills and enhance own readiness for effectively design and participation in virtual mobility.

The process of co-creation of learning resources and activities is driven by theory of Universal Design for Learning (UDL), which is a used as a guide to create meaningful learning experiences that are appropriate for a diverse group of learners [20]. Using UDL principles allows to design for diverse needs and interests of students by creating possibilities to demonstrate how learners have met learning outcomes, e.g. by providing evidence as part of the e-assessment. In order to take the diversity of learners’ needs and interests into consideration, three strategies are recommended by [20], i.e. (a) presenting content in different ways (the “what” of learning), (b) providing different activities for the learner (the “how” of learning), and allowing different paths for learners to achieve goals and to make a meaningful connection to the activity (the “why” of learning) [15]. One practical way to support the process of co-creation of learning resources and activities by means of meaningful gamification is to allow users to set and/or customize their own goals within the VM Learning Hub: “The design challenge here is to support and guide the user in setting long- and short-term goals such that they become achievable and provide experiences of mastery on the way” [14, p. 37].

As any meaningful game and gamification requires not only freedom of users but also the constraints of the game, the approach of allowing learners to set and/or customise own goals also includes constraints which are placed upon learners’ choices and in this way provide guidance toward making choices that are both meaningful to the learner and that meet the overall goal of the VM Learning Hub which is the enhancement of VM skills as a way to enhance readiness for the uptake of virtual mobility in higher education. The VM Learning Hub is thus designed as a flexible system which allows learners to choose from a variety of options and creates possibilities for customization. The meaningful design of the VM Learning Hub allows learners to (a) customise learning goals based on a predefined set of possible learning goals within the hub and helping learners understand how the goals of the activity connect to personal goals such as developing a particular set of virtual mobility skills, (b) choose from available open credentials which are used to mark the achievement of specific pre-identified skill-sets, which are also meaningful outside the VM Learning Hub, (c) provide own evidence to demonstrate how the learning goals/outcomes have been met as part of evidence-based assessment, (d) provide peer-assessment of the evidence attached to open credentials, (e) choose from and develop own ways of engaging with the activities such as co-design of OER, (f) create own activities and transform existing activities to match own goals and needs (e.g. defining typical obstacles in implementing and/or participating in virtual mobility and designing and/or transforming activities to overcome these obstacles (which leads to feeling satisfied and positive about own abilities by overcoming meaningful obstacles), (g) match learners into optimized groups based on individual criteria (e.g. similar goals, complementary prior knowledge, suitable combinations of personality traits for group work) so that learners can find meaning through group engagement, and (h) share the content with other users and non-users of the VM Learning Hub (e.g. colleagues and students in own organisations or in other organisations, e.g. via social media).

The approach to designing learning activities and meaningful gamification in the VM Learning Hub is in line with the theory of user-centered design which for every design decision asks and answers the question How does it benefit the user? and considers user needs and goals at every stage of the design process [15]. This approach is also in line with the theory of situated motivational affordances, which is a Human-Computer Interaction approach to gamification emphasising the need to consider not only artifact affordances (e.g. affordances of gamification elements) but also situational affordances (e.g. affordances of the organisational or learning context) as factors influencing a successful user interaction in a gamified context [22]. The expectation is that the user-centered meaningful gamification design as described above will result in a deeper engagement of learners and in consequence in enhanced readiness for virtual mobility.

4.2 Matching Mechanisms for Collaboration in Groups

In pedagogy, core benefits of group learning are widely agreed upon, like deeper insight into the topic, better argumentation skills and improved social competencies due to applied peer education concepts in the group learning process. Nevertheless, even in small learning scenarios teachers are overwhelmed to form optimized learning groups for certain learning goals. Consequently, mostly learning groups are formed randomly or by self-selection, while it is well known that these approaches have major drawbacks [23]. In pure online learning, like the VM Learning Hub, students and teachers may not know each other. In such learning environments, but also in larger classrooms, an algorithmic approach to form optimized learning groups is needed.

From the algorithmic perspective it is only possible to find near-optimal solutions as the brute-force calculation of optimal learner combination in groups is of complexity class Ω(αM) with α > 1 and M as the number of participants [24, p. 646], [25, p. 90f]. Some approaches use machine learning techniques based on prior behaviour of individuals and preceding group work results. Their advantage is the incremental improvement for one certain learning goal for a specific cohort of learners. On the contrary the causality remains unclear and it cannot be didactically explained why such groups perform well. Thus the results are not transferable to other learning goals, cohorts of learners or tasks. A second group of approaches uses agent-based systems where digital agents represent learners and negotiate with each other for learning group formation. While their strength is the continuous, non-discrete dynamic formation and re-formation of groups based on changes in values the agents optimize their position for, they do not support consideration of global criteria, like similar group quality of size of groups in the while cohort. A third type of approaches is based on numerical optimization with boundary conditions and uses learner criteria vectors to find near optimal combinations satisfying a target formula for learning goals. While they can only be applied at discrete points of time to group all learners, they allow consideration of boundary conditions of the groups and the whole cohort. A more comprehensive overview of algorithms can be found in [26] or [27]. One flexible algorithm with a numerical approach is GroupAL [28] which proved already its benefits in simulations and field studies and is currently the best candidate to be applied for learning group formation in the VM Learning Hub.

The question which criteria are the most relevant ones for learning group formation, is as important as the algorithmic approach. While machine learning approaches tend to use rather unstable criteria which depend on a certain learning scenario and task types, but are quite easy to measure within a learning management system (e.g. Moodle). Such criteria are for example a degree of participation in discussions or rating of group work quality. Stable criteria, on the contrary, are defined as person-related aspects that do not change significantly when learning scenarios or group compositions switch. Therefore, if positive effects of certain combination of such stable criteria on the quality of learning groups are known, they are more likely transferable and applicable to several task types and learning goals. Which of these stable criteria are candidates for optimized learning group formation is part of the research field pedagogical psychology.

To optimize learning groups in a VM Learning Hub with participants from all over Europe, robustness against cultural differences of the selected criteria becomes a major aspect, which is rarely investigated yet. Only for personality traits, mostly referred to as the Big 5 (i.e. extraversion, conscientiousness, openness, social agreeableness, and neuroticism), it has been proven that these are stable over time for individuals and are independent of cultural backgrounds [29]. As no robust way to measure them indirectly from interaction behaviour with a learning system has been found yet, a robust questionnaire measurement has been defined [30]. Recent studies found for satisfaction with the group work, quality of group work results and cohesion within the group two aspects to be most relevant: each learning should have one member with outstanding level of conscientiousness (to align the group activities to the learning goals) and one member should have an outstanding level of extraversion (to guide the group and initiate activity continuously) [27]. Other heterogeneous or homogeneous combination of factors stayed behind in effect sizes. Still, for group cohesion, the amount of time per week, participants are willing to invest, should be matched homogeneously within a group to prevent dissatisfaction. In case when learning scenarios afford synchronous tasks to be performed by groups, even the weekly schedule could be considered.

While the mentioned criteria are found to be relevant in related literature and prior studies, the formation of learning groups for VM skills has not yet been investigated. Thus, all active members of the openVM project participate in collection, prioritization and literature research on additional criteria to consider for optimized group learning of VM skills. It is expected that no commonly agreed and proven set of criteria can be found. As a consequence one or more sets of optimization criteria will be selected based on the relevant literature base. Learners using the VM Learning Hub will be informed about the scientific research aspect of the learning group formation and they offered to opt-in for optimized learning group formation. To allow participation in group activities without agreement, participants will be grouped randomly or by self-selection.

4.3 E-Assessment of Virtual Mobility Skills

Beside the group formation tools, the e-assessment tool will be implemented in the VM Learning Hub. The main role of e-assessment will be to assess virtual mobility skills. This can take many forms including automatic self-assessments and human-supported evidence-based assessments. By using the e-assessment tool, both individuals and organizations will be able to assess and analyze the skills required for virtual mobility and will be able to identify solutions for developing those skills.

The e-assessment of VM skills is planned to be composed of two main parts: the e-assessment concept and the realization of the assessment tool. The e-assessment concept will be built in 3 phases, i.e. (1) defining the objects of the evaluation with the guiding questions in this phase being: What will be evaluated? Who will be evaluated?, (2) defining the purpose of assessment (e.g. formative, summative, diagnostic), and (3) defining the tools to be used which best suite according to the results of the former two phases. The main objects to be evaluated in phase one are the VM skills. These skills are defined in the first part of the project as part of the conceptual framework though applying different research methods such as Group Concept Mapping as described in the sections below. The framework will also define which VM skills are relevant for which target groups. According to the literature there are different type of e-assessment depending of the purposes: formative, summative and diagnostic [31]. Formative assessment is used to provide feedback during the learning process. Summative assessment provides a quantitative grade and is often conducted at the end of a unit or lesson to determine that the learning objectives have been met. Diagnostic assessment is a form of pre-assessment that allows a teacher to determine students’ individual strengths, weaknesses, knowledge, and skills prior to instruction. It is primarily used to diagnose student difficulties and to guide lesson and curriculum planning. It can also be used after the instruction to evaluate the efficiency of learning. Diagnostic assessment will be used a priori to evaluate VM skills. It will be combined with the Open Credentials issued as part of the project summative e-assessments will be designed as part of the evidences users of the VM Learning Hub have to provide to be awarded with Open Credentials, such as VM skill badges. In the final phase, two categories must be considered for the definition of suitable e-assessment tools. These two categories are: (1) e-testing for automated e-assessment (e.g. based on the test score), and (2) e-assessments with evidence for non-automated e-assessment, i.e. e-assessment which requires the intervention of a teacher or a peer and allows for a deeper conceptual rating and more individual feedback for deep-learning (Table 2).

Table 2. Categories of e-assessments applied in the VM Learning Hub.
Full size table

To support e-testing, a self-evaluation tool will be integrated into the VM Learning Hub. This tool is already used in another project (self-assessment tool in elene4workFootnote 4). The tool consists of a survey with a Likert scale which allows users to self-evaluate their soft skills by answering a set of questions which have been defined in the competency framework in the same project. To support the e-assessment with evidence, evidence-based tools, like assignments or e-portfolios, will be offered to enable learners to demonstrate their VM skills in more qualitative ways. The evidence will be evaluated by peers or teachers and may be attached to Open Credentials such as Open Badges to form a metadata-based digital certificate. Such assessment will be useful for the individuals and organisations to adapt their practices to improve VM skills. They can also help learners to choose OERs and MOOCs recommended by the VM Learning Hub to improve less developed skills. According to the results obtained from the three-phase of e-assessment design, a comparative study will be launched to collect different e-assessment examples. The higher education community, including project partners, will provide different examples of e-assessment forms for skill assessment. The goal is to arrive at least one e-assessment example suitable for each category of e-assessment (e.g. e-portfolio, survey, quiz) and for each type of assessment (formative, diagnosis or summative). The collection of examples will be done through an online template and will provide helpful material to build the e-assessment to evaluate VM skills. The template will include several items showing how skills are evaluated (e.g. in which forms and formats and for what kinds of skills). With the skills defined for the openVM framework, the analysis of the collected e-assessment forms and the VM Learning Hub concept, the e-assessment tools for the VM Learning Hub will be designed, implemented and tested. The e-assessment tools have to respect technical specifications defined for the VM Learning Hub as described above. The e-assessment tools must also meet quality criteria other than technical criteria such as reliability, validity and objectivity [32]. One of the most important quality criteria for e-assessment is the validity, including the construct validity, which in this context refers to whether an e-assessment measures the intended construct. In order to optimize this criterion, the evaluations will be based on the principle of scoring grids, i.e. rubrics which are scoring guides, usually in the form of a matrix or grid, with criteria and quality definitions for these criteria. A rubric for e-assessment will be applied as a tool used to interpret and evaluate learner skills demonstrated in evidence-based e-assessment against the criteria and quality standards specified in the grid. To enable users of VM Learning Hub to benefit most from the e-assessment tools, guidelines will be realized and implemented within the tools. These will also give advice to users (e.g. peers, teachers) about how to interpret the data in form of a graphic results with comparison to rubrics and/or other users means values.

4.4 Open Credentials for the Recognition of Virtual Mobility Skills

Open Credentials will be used as a component of the VM Learning Hub to recognise virtual mobility skills. Open Credentials encompass various tools and approaches, including Open Badges and Blockcerts.

Open Badges for Recognition of Skills:

Open Badges are the emerging standard to digitally valorize informal and non-formal learning, or to “communicate skills and achievements by providing visual symbols of accomplishments packed with verifiable data and evidence that can be shared across the web” [33]. Initiated by the Mozilla Foundation in 2011, the Open Badge standard has been now adopted worldwide by individuals and organisations including higher education institutions to look for ways to give proper value to specific competencies, which often remain unrecognised and/or are not made transparent by formal degrees and certificates. Open Badges have a form of a Portable Network Graphics (PNG) with embedded metadata allowing the verification of the Open Badge validity and ownership [34]. Open Badges provide a digital, open and flexible way to define skills and competencies, identify them visually and issue a proof of competency mastery to learners which can be accompanied by specific evidences and outcomes, and is always completed by a clear description of criteria that needed to be met to earn that Open Badge. While Open Badges have notably been used to provide a digital, readable and quickly verifiable evidence of formal degrees (e.g. University of Milan BicoccaFootnote 5 and University of BicoccaFootnote 6), it is significant that the most enthusiasm has been drawn to the field of soft skills, especially important for an ever changing contexts and lifelong learning [35, 36]. Soft skills, which comprise a large part of the virtual mobility skill-set, are also difficult to identify [37]. The problem of agreeing on competency definitions will be avoided by making the definition and the alignment to the competency frameworks used to define VM skill-sets transparent. The same will be done for criteria: each Open Badge issuer will be able to define own criteria, and Open Badge readers will be able to choose how an Open Badge should be evaluated based on those criteria and the related evidences. The openVM project will apply the Open Badge concept to the Virtual Mobility domain, leveraging the BestrFootnote 7 platform developed by one of the project partners. The project will identify which competencies would benefit from being represented with an Open Badge, also with reference to the user experience and the gamification approach that the use of Open Badges enables. Having a visual representation of a goal to be achieved by collecting a series of learning experiences or evidences is of course a basic element for gamification, but an extremely important added value for all learners is represented by the fact that such achievement will also be valid outside the learning experience and can actually be inserted in CVs, where it will provide instant verification not only of the validity of the Open Badge but also of the criteria and institution who issued the badge. The VM Learning Hub will therefore represent relevant examples of institution (hub of institutions) providing assessment in the soft skills field, an activity currently more required than performed given the intrinsic complexity of evaluating such skills. Lastly, it must be noted that Open Badges can also contribute to a formal learning path being recognised by a school or university, action made simpler by the digital status of the Open Badge as shown by the experience of automatic recognition of badges by the Student Information System [38].

Open Assessment Though xAPIs and Learning Record Store:

The key to the recognition of skills developed in different contexts is the ability to gather learning information from a variety of sources where learning and assessment can happen: instead of designing – with a top-down approach typical of formal learning – a single main learning path where learning experiences should happen, multiple and diverse learning experiences need to be recognised and read in the light of relevant competencies they are developing. Technically, this can be achieved through the use of the Experience API, more commonly called Experience API (xAPI). The xAPI is an open specification designed to allow the interoperable exchange of learning and performance activity data between various systems and applications [11]. The xAPI standard aims at opening up the way learning experiences can be captured, stored and used. With xAPI any system can express a learning experience through a statement describing how a learner (the subject) has performed an action (the verb, e.g. answered), with reference to an object (e.g. questionnaire). Statements are captured in xAPI format and stored in a Learning Record Store that acts as a specialized database for xAPI data generated by different systems (e.g. LMS, apps, blogs, forums), called activity providers. The Bestr platform, used in the openVM project to provide Open Badges, has its own Learning Record Store and is capable of collecting xAPI statements from any platform integrated with it [39, 40]. Criteria for issuing Open Badges in Bestr are defined according to verbs and objects of learning statements which the learner must accomplish in order to gain a specific Badge. When the platform identifies that a set of statements for a given subject (the learner) is matching the criteria required for a badge, the Open Badge is issued to the learner. Having obtained an Open Badge is a new learning statement (a new learning achievement), which can be used as a starting point for issuing a new Open Badge. Leveraging this system based on open standards, the openVM Learning Hub will be able not only to express its own learning statements connected to the use of its own OER, but also to convey learning statements from any other compliant and authorised platform towards the LRS, and use any set of such information to activate the automatic issuing of a Badge and perform useful analytics for the project.

Blockcerts for Encrypted Certification:

The openVM project will also evaluate the opportunity to introduce, besides Open Badges, the Blockcerts as a Blockchain-based technology for digitally certifying skills. Blockcerts, in the words of JRC report Blockchain in education [41] can be defined as follows: “The cornerstone of the Blockcerts open standard is the belief that people should be able to possess and prove ownership of their important digital records. [..] Within this context, the Blockchain is considered to be a technology that allows individuals to own their official records and share them with any third-party for instant verification, all the while precluding any attempt to tamper with or edit the records.” The openVM project will consider the benefits of a permanent and encrypted recording of achievements such as those that will happen through the VM Learning Hub, as well as costs related to integrating the technology into the VM Learning Hub and to using it on a publicly available Blockchain, where writing transactions requires a constantly increasing fee.

4.5 Competency Directory for Referencing of Virtual Mobility Skills

In the preceding chapters major components of the targeted VM Learning Hub and there benefits based on literature research and practical evidences were described. Some of them will benefit from an alignment to competency definitions. This especially refers to the e-assessment and to open digital credentials which will be both combined with the competency directory to identify certain levels of skills in different domains in order to personalize the learning experience and recommended suitable resources. The freely available resources (such as VM MOOC, OERs) are the second part benefiting of alignment to competency definitions in order to express the skills intended to be learned while using the resources. The major benefit of adding metadata to such components of the VM Learning Hub is the algorithmic ability to recommend more suitable resources for identified skills to be improved, adapt the learning paths more flexible and after passing a certain assessment for competencies to be awarded directly with (all) Open Badges which need (a subset of) these competencies as criteria a learner has to evidence. The Open Bade Specification [42] allows the use of extended Uniform Resource Locators (URLs) as Internationalized Resource Identifiers (IRIs) to avoid ambiguities in referencing the same competency in different Open Badges (e.g. languages versions align to the same competency). Specific data formats in the VM Learning Hub will need yet to be decided to support IRI-based referencing. While it is already a great benefit to have unambiguous IRIs for competencies that allows algorithms to easily detect same reference to same competencies, it would be of additional advantage to be able to identify levels of competencies, similarities and prerequisites as well as successors of competencies. Thus, web servers, hosting competency definitions, deliver a response in a machine readable format via Hypertext Transfer Protocol (HTTP) as answer to a HTTP client request for the IRI. While not yet one open format is established, JavaScript Object Notation-Linked Data (JSON-LD) or Resource Description Framework (RDF) are promising candidates. Beside information about competencies themselves (name, description, translations, levels) they allow cross-referencing other IRIs by semantic tuples (resource, property, value), e.g. IRI1 is part of IRI2 where IRI{1|2} represent competencies. The same concept is used to store activities in xAPI as described above.

While manifold competency frameworks exist, each is defined for a certain (narrow) domain and only a few provide unique referencing to competencies based on IRIs. Even worse, most existing competency frameworks do not provide machine readable formats, but publish their competency definitions as PDFs containing self-made IDs of competencies. To solve these problems for better machine-based processability, efforts exist to centralize and define the one competency framework to contain them all [43, 44]. It can be questioned that such efforts will ever succeed as competency definitions change over time, new competencies arise due to evolution and others fade. Additionally, different cultural backgrounds add ambiguity, e.g. for competencies like cooperativity [43]. Consequently, a more feasible goal is to keep and maintain manifold competency frameworks but cross-reference among them in a meaningful, semantic format to allow algorithms to deduct similarities and detect connections (i.e. paths) from one competency definition to the other. Thus, VM Learning Hub will provide its own hosted competency framework containing the definitions of all VM related skills needed for the hub. These are to be provided in a machine-readable format via a Representational State Transfer Application Programming Interface (REST-API) using unique IRIs per competency. Beside the format (JSON-LD or RDF) the vocabulary to express semantic data needs to be decided. While InLOC [45] is an open and well-defined vocabulary for skills, competencies, knowledge and evidences of competencies, it lacks implementation in practice. Since 2017 the European Union encourages the use of European Skills Competency and Occupation framework (ESCO) vocabulary, which is less precise in abilities to express competency frameworks but a first implementation exists containing more than 8000 competencies and their relations as used within the European union for job descriptions [44]. In case of using ESCO vocabulary it is an optional add-on to cross-reference VM Learning Hub competency definitions to similar or superset competencies defined by ESCO. The vocabulary of Competency and Skills System (CASS) defined since 2015 is less complex in variations of expressible relations than ESCO, but provides Open Source licensed client and server applications with the ability to decentrally host several interconnected CASS instances [43]. First pilot additions of ESCO defined competencies exists. In case requirement analysis for the VM Learning Hub competency directory reveals editing and cross-referencing needs, using CASS as a basis might be suitable. Developed in the former project Open Badge Network EuropeFootnote 8, Competency Directory supports the automatic crawling and updating of competency framework data and provides unique URLs with a web-based search interface [46]. It was designed for light-weight searching for IRIs of competencies to use them in Open Badge alignment and criteria fields. Beside CASS it is a valuable alternative code base to be used, but offers no front-end editing yet as it automatically fetches competency frameworks from their servers.

5 OpenVM Study and Design Implications

5.1 Aims and Methods of the OpenVM Study

As the experiences with and knowledge of the concepts of Virtual Mobility and Open Education vary greatly, depending on individual implementations, one of the first steps in the openVM project is to develop a shared understanding of the concept of Open Virtual Mobility and to establish the core characteristics of Virtual Mobility activities against the backdrop op Open Education. This is achieved through a series of research studies leading to establishing the conceptual framework. Given the focus of the openVM project on credentialing learner skills, a particular focus was placed within this study on the learner skills and competences that can be developed in Open Virtual Mobility contexts. For this purpose, the Group Concept Mapping (GCM) methodology was applied. GCM supports knowledge construction through collecting and organizing ideas of individuals so that a visual geography of a concept can be created to be further analyzed, interpreted and used to feed understanding, design and/or decision or policy making [47, 48].

The GCM follows several distinct phases, in which all or a selection of participants take part. It starts with idea generation which can be based on the input of different stakeholders and/or literature reviews. Idea generation is followed by involving participants in organizing collected ideas and evaluating them on a number of relevant dimensions, e.g. importance and feasibility [49]. Thereafter, the input is analyzed with two advanced multivariate statistical techniques - multidimensional scaling (MDS) and hierarchical cluster analysis (HCA) to identify patterns in the data. The output of this analysis are maps representing collective standpoints on an issue or a concept under research together with individual positions. Such maps can be used to validate the shared understanding and to formulate further actions or strategies [47].

In order to understand the views on learning in an Open Virtual Mobility context a range of stakeholders, including representatives of research community as well as educators, students, internationalisation officers, higher education leaders and policy-makers were invited to participate in a GCM study. All project members of the openVM project contributed as experts on both Virtual Mobility and Open Education. The majority of participants were university professors and/or researchers with at average 20 or more years experience in education and affinity with the concepts of Virtual Mobility and Open Education. In total 101 statements were generated in phase 1 of the study. Statements were checked whether they contained repetition or ambiguity issues by two project members. All duplicates were removed. The final 90 statements were grouped by study participants based on their similarity in meaning, provided with meaningful labels and evaluated on dimensions of importance and feasibility by the first nine representatives of the VM expert community (initial stage). While the participation of more experts both from practice and from the research community is to follow, first results of the GCM study provide insights into the way experts understand VM skills which form a basis for further development of the VM Learning Hub and its components.

5.2 Preliminary Results of the OpenVM Study

The preliminary results of the GCM study are based on the clustering and rating of the 90 statements by the first 9 participants. The point map of the 90 collected statements has the stress value of 0,333 which indicates that this map is a good representation of the raw data [50]. Based on a combination of cluster analysis suggested by the tool and a discussion of the clusters by the two researchers, a nine cluster configuration was selected. Bearing in mind, that this is preliminary data based on the input of 9 first participants, the configuration as presented in Fig. 2 was considered sufficiently differentiating and meaningful. The preliminary character of the results is, however evident: the clusters differ in average bridging values of constituent statements from extremely low (0,001) to extremely high (0,99). The bridging values (from 0 to 1) point to the extent of coherence in the way individual participants group statements. A low bridging value indicates that participants agree relatively easily on the content of the cluster. A high bridging value means that a statement has been grouped together with statements farther apart.

Fig. 2.
figure 2

Cluster map with statements from the preliminary phase of the openVM GCM study.

Full size image

Only one cluster, Intercultural Literacy, has a really low bridging value, indicating a great deal of agreement between the 9 participants over the underlying statements. We can therefore deduce that this cluster even at this preliminary stage of the study points to a clear learner skill that can be supported and developed through Open VM. Table 2 shows an overview of the statements in this cluster, with their related bridging values and also with average values of ratings on dimensions of importance and feasibility (Table 3).

Table 3. Cluster intercultural literacy and constituent statements.
Full size table

The five clusters have medium bridging values. These are: Interdisciplinary skills (0.34), Programme characteristics (0.35), Intercultural skills (0.38), Motivation and Guidance (0.42) and Learning Techniques (0.47). The remaining 3 clusters Language Skills (0.80), Student Needs (0.70) and Attractions of Programme (0.87) show high bridging values, indicating a higher possibility of coincidence in sorting. Taking into consideration that only 9 participants performed the sorting and rating phases of the GCM, we do not discuss these clusters here. We do expect these clusters to evolve with the participation of more experts.

5.3 Implications of the OpenVM Study for VM Learning Hub Design

Intercultural literacy emerges as a clear learner skill from the preliminary results of the GCM study. A closer look at the cluster reveals that the statements deal with cultural aspects of an Open Virtual Mobility activity, more specifically, about the potential intercultural learning that a learner in OpenVM can experience. Four aspects of intercultural theories are covered in this cluster [51,52,53] (and IERESTFootnote 9 platform): knowledge (74, 75, 23, 10, 15, 43, 40), attitude (61, 62), self-awareness (48, 6, 31) and skills (78, 49) Interestingly, statements relating to gaining knowledge of other cultures show 0 or near-0 bridging values. Statements relating to self-awareness also score relatively low. There is more divergence in opinion within our 9 participants on intercultural skills and attitudes. One key implication from this initial stage of the GCM study for the design of the VM Learning Hub is that it necessarily needs to focus the development of intercultural literacy skills. Further completion of the study will show which other skills are deemed relevant for Open Virtual Mobility.

6 Conclusions

This paper has presented the concept, approaches, considerations and first study results relevant for designing a collaborative learning hub for promoting VM Skills of educators and students in the European Higher Education Area. While the development of the VM Learning Hub to enhance the Virtual Mobility readiness through achievement, assessment and recognition of VM skills is still at an early stage, the authors of this paper aimed to demonstrate the complexity of designing such as collaborative learning hub with the view of helping in planning and decision-making in similar projects. The considerations presented here may be interesting for other projects and contexts which aim to apply technologies for collaborative forms of skill attainment, skill assessment and skill recognition.