1 Open Educational Resources as a Component of Digital Learning Ecosystems

Times when learning processes took place via a central location or channel are increasingly becoming a thing of the past. Instead, in the digital age, the focus is shifting to digital learning structures that support the acquisition of new knowledge or new skills. New technologies, technical infrastructures, and software systems provide the basis for these learning ecosystems. They offer space for content and content formats and support the usage processes. Users play an important role within such learning ecosystems: they are learners, consumers, designers, critics, and providers of impetus and content. By integrating them into the processes and interconnecting them with one another, an active, lively learning culture is created. Through the rapid development of new content, the resulting learning content adapts to new (continuing education) needs, resulting in agile competence acquisition. (Hofschröer et al., 2019).

Because of their free usability and redistribution, Open Educational Resources (OER) offer great potential in this context. In its definition, which was revised in 2015 and is still valid today, UNESCO defines OER as

“[…] any type of educational materials that are in the public domain or introduced with an open license. The nature of these open materials means that anyone can legally and freely copy, use, adapt and re-share them. OERs range from textbooks to curricula, syllabi, lecture notes, assignments, tests, projects, audio, video and animation.” (UNESCO, 2015)

Due to their mostly digital format, OER facilitate agile and formative networking and enable learners to play an active role in educational processes.

1.1 Quality Assurance of OER as a Challenge

However, with free usability and redistribution come new challenges for the quality assurance of open teaching and learning resources. Hirsch et al. (2016) state that, along with copyright issues, the question of the quality of open educational materials is one of the most discussed topics in the context of OER and is central to the acceptance and success of free educational materials.

Yuan et al. (2015) point out that given the growing number of free educational materials available on the Internet, identifying high-quality resources is becoming increasingly difficult. Furthermore, it should be considered that users often evaluate the quality of OER based on personal quality requirements and their own needs, so that quality requirements may vary individually. According to Ehlers (2015), a particular challenge for the quality assurance of OER is the processual nature of the materials. Traditional quality assurance processes establish certain criteria and then check whether these criteria are met. For OER, these static quality checks are only partially effective. Yet it is precisely this openness and adaptability to individual needs that requires special attention when it comes to ensuring the quality of OER: if materials are developed predominantly based on individual needs and subjective criteria, it can be difficult to identify high-quality OER. Here, a quality concept that provides users with at least a basic framework of quality standards could be helpful to make the quality of OER measurable and to strengthen trust in open educational materials.

Distributed learning ecosystems, as described by Otto and Kerres (in this volume), pose special challenges for the quality assurance of OER. Users of OER who access linked repositories via such an infrastructure are often confronted with a multitude of resources on a given topic.

Moreover, these resources do not originate from a single OER portal but from a wide variety of linked sources. Users therefore hardly have the possibility to evaluate the quality of the resources based on their origin. On the one hand, the reputation of the source as indication of the quality of OER is missing, and on the other hand, it is not feasible for users to evaluate the quality standards of a large number of linked repositories and compare them with their own demands.

2 Approaches to Ascertaining Quality in OER

To date, there is no generally accepted procedure or approach for quality assurance of OER. However, the lively discussion around this topic nationally and internationally has led to the development of a variety of different approaches and ways to measure quality in open educational media and to ensure quality assurance. In the following, selected methods are presented.

2.1 Procedures for the Quality Assurance of OER

Many of the approaches to quality assurance of open educational materials relate to their use in the school context.

For example, the base initiative and online platform “Zentrale für Unterrichtsmedien im Internet e. V.”Footnote 1 (ZUM) relies on a wiki structure for the provision of material, which enables continuous improvement and error correction through comment functions and revision suggestions.

Another method is being tested by the “edutags”Footnote 2 initiative, which offers a collective tagging system for educational materials on the Internet. The idea behind this is that peer-to-peer tagging makes high-quality materials easier to find.

The Augsburger Analyse- und Evaluationsraster für analoge und digitale Bildungsmedien (AAER)Footnote 3 offers the possibility to evaluate a teaching–learning medium, to participate in group evaluations of an educational medium, and to view already existing evaluation results. A questionnaire examines a total of eight dimensions (curriculum and educational standards, discursive positioning, macro- didactic and educational theoretical foundation, micro- didactic foundation and implementation, cognitive structuring, picture and text composition, task design, application transparency). The structure of the AAER allows the teachers applying it a differentiated view of the strengths and weaknesses of educational media in the different areas (Fey, 2015, 2017).

Various approaches and ideas for the quality assurance of OER are now also available for the higher education sector.

The Hamburg Open Online University (HOOU)Footnote 4 promotes the creation of scientific digital learning opportunities and offers free-to-use learning opportunities on its educational platform. The primary basis for HOOU’s framework of quality assurance is an evaluation procedure, which serves a supporting and consulting function. The editorial reviews, however, exclusively concern the pedagogical-didactical and technical areas; the content design is the responsibility of the authors. (Friz, 2019). The HOOU is currently developing a questionnaire to check the quality of its OER.Footnote 5 It will examine four different pedagogical-didactical and technical dimensions: content (scientific foundation, target group orientation, reusability of content), didactical design (alignment, collaboration and interaction, application and transfer, assistance and support, assessment), accessibility (CC license, accessibility for people with disabilities, reliability and compatibility, technical reusability) and usability (structure, navigation and orientation, Interactivity, design and readability). As a long-term goal, HOOU has formulated a “brand core” that aims at both, the free and open licensing of OER and the development of an HOOU label for OER as a quality seal. (Zawacki-Richter et al., 2017).

The Lower Saxony OER portal “twillo”,Footnote 6 funded by the Ministry of Science and Culture of Lower Saxony (MWK), promotes the establishment and expansion of a sustainable infrastructure for the provision of OER. The quality assurance procedure primarily involves an assessment of the materials by the creators themselves. The OER quality check offers direct assistance to users by guiding them through a questionnaire comprising seven dimensions (content reusability, design and readability, structure and orientation, scientific foundation, motivation, assistance and support, application and transfer). In addition to the OER quality check, the twillo portal offers a constantly growing collection of advice and assistance on the topic of quality assurance of OER.

In summary, the presented approaches to quality assurance of OER show an inconsistent picture. As a rule, the responsibility for the quality of OER lies primarily with the authors who publish free educational materials. A number of procedures aim to involve users directly in the quality assurance process. For this, wiki structures, commenting and tagging systems, evaluation, revision, or peer review processes are used. In addition, the OER platform operators provide service and consulting offers as well as editorial support services.

However, a widely accepted model and instrument for quality assurance of OER does not exist. This unsettles many (potential) users. Especially against the background of the increase in digital learning structures and the free usability and dissemination of open educational materials, it is necessary to develop a quality instrument that adequately considers the characteristics of OER (i.e., reusability, modification, processability). (Brückner, 2018; Zawacki-Richter et al., 2017).

For OER used in higher education, the following is an example of the development of a respective tool.

2.2 A Model for the Quality Assurance of OER in Higher Education

In 2017, Zawacki-Richter and Mayrberger conducted an extensive literature review and identified eight international approaches to OER quality assurance:

  1. 1.

    Learning Object Review Instrument (LORI) (Nesbit et al., 2007)

  2. 2.

    Multimedia Educational Resource for Learning and Online Teaching (MERLOT Rubric) (California State University, 2019)

  3. 3.

    Framework for Assessing Fitness for Purpose in OER (Jung et al., 2016)

  4. 4.

    OER Rubric (Achieve Organization) (Achieve Inc., 2011)

  5. 5.

    Learning Object Evaluation Instrument (LOEI) (Haughey & Muirhead, 2005)

  6. 6.

    Learning Objects Quality Evaluation Model (eQNet) (Kurilovas et al., 2011)

  7. 7.

    Rubric to Evaluate Learner Generated Content (LGC) (Pérez-Mateo et al., 2011)

  8. 8.

    Rubric for Selecting Inquiry-Based Activities (Fitzgerald & Byers, 2002)

The subsequent analysis of the identified quality assurance tools for OER showed enormous differences in terms of their respective complexity and levels of detail. In terms of content assessment, two basic groups were identified: the first group offers simple catalogues of criteria or checklists (Framework for Assessing Fitness for Purpose in OER, Rubric for Selecting Inquiry-Based Activities, Rubric to Evaluate Learner Generated Content), while the second group summarises approaches and tools in which the quality criteria are evaluated on a scale. Here, too, differences become apparent: Some of the analysed approaches are based on a quality model with several quality dimensions, to which a number of quality criteria are assigned (e.g., eQNet, MERLOT), yet, no weightings are given to individual dimensions and criteria. Other approaches consist only of lists of criteria (e.g., Achieve). There are additional differences. For example, while the LORI instrument offers a detailed scoring guide for operationalising the rating scales, other instruments consist of simple checklists (e.g., Framework for Assessing Fitness for Purpose in OER). In terms of the context of use, there are generic approaches as well as those developed for a specific subject domain, for example, science (Rubric for Selecting Inquiry-Based Activities), schools (e.g., LOEI), or user-generated content (LGC). (Zawacki-Richter et al., 2017).

It is striking that the number of evaluation criteria varies greatly. They range from eight (Achieve, eQNet) to 42 (LGC). In total, 161 criteria are used in the eight evaluation instruments. To assign these 161 identified quality criteria to a system for a synoptic summary, Zawacki-Richter et al. (2017) extended the quality model of Kurilovas et al. (2011), which was developed within the eQNet Quality Network for a European Learning Resource Exchange. Using this criteria model, they summarised the various quality indicators in the form of a conceptual tree (e.g., Fig. 1). The so-called IPR (Intellectual Property Rights) criteria stand out here, as they are explicitly listed as the main dimensions regarding OER, alongside technical and pedagogical-didactic criteria.

Fig. 1
The summarized view of a page has quality indicators of the technological criteria of intellectual property rights.

Indicators for quality assurance of OER (own translation; in accordance with Zawacki-Richter et al., 2017, p. 45)

Full size image

Based on the indicators found, Mayrberger et al. (2018) developed their own model of the quality of OER, which included 15 quality dimensions assigned to the four areas of “content”, “instructional design”, “accessibility”, and “usability” (Fig. 2.).

Fig. 2
The block diagram of open educational resources branched off into the pedagogical and technical dimensions to provide high-quality assessment results.

OER quality model (Mayrberger et al., 2018, p. 29, translated by the authors)

Full size image

The model thus describes quality as a complex, multi-faceted construct.

It includes content-related and pedagogical-didactic as well as technical dimensions. Thus, it contains quality criteria that are specific to the demands of the higher education sector as well as requirements that result from the specific dynamics of the development of OER. While the content-related quality criteria ensure that OER meet the expectations of usage in the higher education context, the technical dimensions in “accessibility” enable an open further development of resources as well as the adaptation to specific learning contexts and requirements.

The consideration of “accessibility” quality dimensions guarantees that the dynamic development of quality in the sense of Ehlers (2015), which is typical for OER, is rendered possible.

Most of the individual dimensions in the model also form complex constructs themselves. To make quality validly assessable and measurable, these individual constructs must also be operationalised by objective, reliable, and valid scales. Educational measurement and the associated test and measurement theory are concerned with the development of such scales. Within the framework of the EduArc project, scales for the model of Mayrberger et al. (2018) should therefore be developed and validated using the procedures of test and measurement theory.

3 Test Theoretical Development and Validation of an Assessment Tool

3.1 Instrument for the Quality Assurance of OER (IQOER)

To capture the quality of OER, Mayrberger et al. (2018) propose IQOER, an instrument consisting of a long and a short version. In the short version, each of the 15 dimensions of the quality model of Zawacki-Richter and Mayrberger (2017) is operationalised in the form of a 5-level classification scale (Mayrberger et al., 2018; Fig. 3).

Fig. 3
A five-level classification scale represents the quality assurance of O E R in five levels as reference origin of models, methods, and coherent reasoning.

IQOER (short version): Classification scale “Scientific foundation”; own translation

Full size image

The dark green level of the classification scales is referred to as the “premium standard” (Mayrberger et al., 2018 p. 21) It should be chosen for resources that are “significantly above expectations”. The medium green level (4) denotes resources that are “significantly above expectations”, and the light green level indicates the “minimum standard”. The lower two levels denote (to varying degrees) failure to meet the minimum standard.

The red, light green, and dark green levels of the classification scales are all described by several statements (descriptors). The intermediate second and fourth levels are not described, so it is up to the raters to interpolate the content of these levels from the other levels.

The assessment of characteristics using individual classification scales is associated with two problems from the point of view of measurement theory:

First, if a characteristic is only determined using a single rating, split-half reliability or internal consistency cannot be determined because there is no other measure to correlate with.

Secondly, and more importantly, a classification scale forces a joint evaluation of possibly incompatible statements. Each classification scale (Fig. 3) consists of several statements that do not necessarily have to be equally true for a particular resource. For example, a resource may well have consistently cited bibliographic sources, but the reasoning within the resource may not be coherent. In such a case, in the example from Fig. 3, the rater is faced with the difficulty of deciding whether the dark green option is right. Ultimately, the rater is forced to weigh the different statements arbitrarily and make a rating accordingly.

For these reasons, classification scales have a very low usage in the empirical social sciences (e.g., psychology), where a measurement-theoretical foundation of the scales is required.

An alternative to classification scales is to average across scores from different individual items. Such items consist of a single statement to which the rater expresses agreement or disagreement on a multipoint Likert scale. The items of a scale are combined mathematically, using classical test theory, by simply taking a mean or sum of the individual ratings (e.g., Wu et al., 2016). If items of a scale have opposite content orientations, they have to be recoded before averaging (e.g., on a 5-point scale, 1 becomes 5, 2 becomes 4, 4 becomes 2, and 5 becomes 1).

Such ratings using Likert scales require raters to make only simple judgments on clear statements about resources. Figure 4 shows the scale “Scientific foundation” operationalised based on five individual items. In this case, the scale is formed by the mean of the item ratings. The alternative “does not apply at all” is coded as 1, “fully applies” as 5, and the alternatives in between are coded as 2 to 4. Items with opposing content (e.g., item 2 in Fig. 4) are recoded.

Fig. 4
A table consists of theoretical development results of open educational resources as the content relevancy, precision, coherency, and many more.

Recording of the scale “Scientific foundation” using individual items based on (Mayrberger et al., 2018, S. 35)

Full size image

Therefore, using the classification scales to operationalise the model by Zawacki-Richter and Mayrberger (2017) serves as a short version of the instrument, whereas the assessment by using individual items serves as a long version. The instrument is referred to as the Instrument for Quality Assurance of OER (IQOER).

3.2 Empirical Validation and Optimisation of the Instrument

The IQOER instrument was empirically tested and optimised in a multi-stage validation study. For this purpose, in a first stage, approximately 50 raters rated 2 OER each using the instrument. The ratings were used to determine interrater reliabilities, internal consistencies of the item scales, and other parameters. Since both, item scales and classification scales were surveyed, convergent and discriminant validities could be determined regarding the two forms of data collection.

To determine the construct validity of the IQOER instrument, raters also assessed the OER using the MERLOT Peer Review Form (California State University, 2019), another instrument for OER quality.

The aim of the empirical study was to obtain suggestions for a revision of the instrument that would allow optimisation regarding test-theoretical quality criteria (i.e., reliability and validity). For this purpose, procedures of the classical test theory were used (Wu et al., 2016).

The validation study is part of an effort to develop a German language instrument to assess the quality of OER. Therefore, all instruments and resources used were in German.

The study was conducted as part of the joint project “Digitale Bildungsarchitekturen—Offene Lernressourcen in verteilten Lerninfrastrukturen [Digital Educational Architectures—Open Learning Resources in Distributed Learning Infrastructures]—EduArc”, funded by the German Federal Ministry of Education.

3.2.1 Sample

A total of 50 raters participated in the study. Each rater had the task of assessing two different resources. One rater assessed three resources. In total, 101 resources were assessed. Eight different resources were each rated once, 33 resources were rated by two raters, and nine resources were rated by three raters.

76% of the raters reported being female, 20% reported being male, and 2 (4%) selected the option “diverse”. Raters had a mean age of 31.4 years (SD = 9,9).

54% of raters studied or worked in a humanities subject; 28% in a law, economics, or social science subject; and 18% in a STEM discipline. Raters from STEM subjects preferentially received OER with STEM content for assessment. Raters from the other subjects received resources with humanities or social science content.

44% of the raters had a master’s degree or a doctorate, 30% had a bachelor’s degree, and another 24% were still studying for a bachelor’s degree.

12% of raters said they had already been involved in the development of OER themselves, and 48% had experience in the use of OER.

3.2.2 Instruments

3.2.2.1 IQOER

For the survey, slightly revised versions of both, the individual items (e.g., Fig. 4) and the classification scales (e.g., Fig. 3) of the IQOER compared to those published in Mayrberger et al. (2018) were available. The present version differs from the published version, inter alia, in the following aspects:

  • An additional dimension, “motivation”, was included. This dimension captures the motivational quality of the resource, the extent to which it is interesting and motivates learners to engage more closely with the content. In the model of Zawacki-Richter and Mayrberger (2017), this scale is assigned to the “Didactics” domain.

  • The terms “learning object”, “learning unit”, and “course” were systematised and unified.

  • Based on discussions with various German OER platform providers, the items were slightly revised and more focused on the requirements of quality assurance of OER.

  • Content inconsistencies between the classification scale levels and items were eliminated.

Only 12 of the 16 IQOER scales were used in this survey. The 4 scales in “Accessibility” require in-depth knowledge of the technical basis of a resource and can, therefore, only be evaluated by technical specialists. Furthermore, these dimensions are only available as classification scales. They were, therefore, not included in the validation study.

Furthermore, not all dimensions of the IQOER can be applied to all types of resources. Five of the selected twelve dimensions involve a pre-selection criterion. For example, for the dimension “target group orientation”, the OER is first assessed with the question: “Does the OER you are looking at contain a reference to a specific target group and/or are required prior knowledge items mentioned?”. Only if the respective criterion is fulfilled, the classification scale and the items are queried. Seven of the dimensions (see Table 2) can be applied to all resources; the scales formed for these dimensions are referred to as the “core scales” of the IQOER (Fig. 5). The results presented here relate exclusively to these core scales.

Fig. 5
A block diagram highlights the critical quality dimensions of open educational resources as 12 I Q O E R and 7 core scales are highlighted.

Dimensions of the IQOER selected for the survey according to the model of (Mayrberger et al., 2018)

Full size image
3.2.2.2 Merlot Peer Review Form

MERLOT (Multimedia Education Resource for Learning and Online Teaching) is one of the most comprehensive and oldest (founded in 1997) OER repositories for higher education in the United States (Malloy & Hanley, 2001; Orhun, 2004). The platform (www.merlot.org) is maintained by California State University in cooperation with a variety of partner institutions and private providers. MERLOT’s quality assurance is based primarily on peer reviews. For this purpose, the platform provides a comprehensive peer review form that is used to evaluate resources (California State University, 2019).

The MERLOT Peer Review Form consists of 31 items in the areas of “Quality of Content”, “Potential Effectiveness as a Teaching Tool”, and “Ease of Use”. Each item is assessed using a five-point Likert scale (ranging from “strongly agree” to “strongly disagree”). A summary overall rating across all three domains is assessed with: “What is your overall numeric rating for this module?”.

3.2.2.3 Assessed OER

This study evaluated German-language educational resources that are published under a CC license and have students at universities or university graduates as their target group. There are 27 OER with humanities or social science content, 7 OER from STEM subjects (science, technology, engineering, and mathematics), and 16 OER with both, humanities/social science and STEM content.

The OER come from relevant public OER repositories (especially from the OER platform of the HOOU Hamburg Open Online University) as well as from commercial platforms such as YouTube.

3.2.2.4 The Online Survey

Raters were given an access code to an online platform. On the platform, they were provided with links to the two OER to assess. They were asked to familiarising themselves thoroughly with the OER and, in case of longer learning sequences, to work through at least 2 chapters of each OER. Immediately afterwards, the raters were asked to access an online questionnaire on the platform. This contained both, the IQOER and the Merlot Peer Review Form in a German online version. Some further items were also collected, which included a general evaluation of the OER.

After evaluating the OER, raters were asked to complete an online questionnaire with demographic questions. For data protection reasons, the demographic data were collected separately and had no link to the OER ratings.

The survey took place from 8/2019 to 2/2020.

3.3 Results

Table 1 shows the 37 items from the 7 core dimensions of the item version of the IQOER (preliminary version).

Table 1 Preliminary items of the IQOER (English translation of German items)
Full size table
Table 2 Pearson correlations of item and classification scales as an indicator for convergent validity
Full size table

The correlations between item and classification scales (i.e., r values) reached a sufficient level for all but two items (item 29 “Contents can be found by means of a search function.” and item 2 “The content of the resource focuses onesidedly on specific providers, products, or models.”). Furthermore, the mean score for item 29 was fairly low at 2.26. The difficulty of this items is very high.

To determine the interrater reliability, an intraclass correlation was calculated between the first and last rating of a resource. Table 1 also shows the interrater reliability of a single rating across all resources. Negative correlations indicate insufficient agreement between raters; this is found in item 15 “In some elements of the OER, at least the first steps of a didactic design are recognisable.” and item 24 “The OER contains a summary of the content presented.”

The Cronbach’s alphas of the 7 item core scales were: α(SCF) = 0,79; α(CRU) = 0,83; α(MOT) = 0,92; α(AAT) = 0,87; α(AAS) = 0,83; α(SNO) = 0,78; α(DAR) = 0,89.

Table 2 shows the correlations of the item and classification scales. The principal diagonal marked in bold can be interpreted as convergent validity, since item and rating scales represent different forms of measuring the same construct.

Furthermore, Table 2 shows the correlations of the IQOER scales with the Merlot total scale. Due to the inverse polarity of the Merlot scale, negative correlations here mean a correlation in the same direction. All IQOER scales have highly significant correlations with the Merlot scale.

Indeed, it turns out that all convergent validities are substantial and highly significant. Within a row or column in Table 2, the correlations on the principal axis are always the highest values. Thus, in all cases, the convergent correlations (i.e., the correlations within a construct measured across different scales) are higher than the discriminant validities (i.e., the correlations of different quality aspects).

Nevertheless, many correlations outside the principal diagonal (i.e., the correlations between different aspects of quality) are also significant. Therefore, the results demonstrate that the different aspects of quality of OER are distinguishable but not independent from each other.

3.4 Interpretation of the Results

The results of this study suggest that the IQOER performed well overall as an instrument for assessing the quality of OER. Both, the short and the long version of the IQOER core scales were able to assess quality aspects of OER that correspond to the quality concept of the Merlot Peer Review Form. This can be interpreted as construct validity for the IQOER.

Furthermore, the item scales all have sufficient internal consistency, which, in some cases, could be optimised with further modifications. In addition, the quality aspects measured have sufficient convergent and discriminant validity (i.e., each of the seven core scales measure separate quality aspects that can be distinguished by the raters). Therefore, the results demonstrate a promising utility of the IQOER instrument.

The analysis at item level also reveals weaknesses of some individual items. Not all items achieved sufficient interrater reliabilities, and some items do not have sufficient discriminatory power regarding the respective scale. The results of the item analysis offer some concrete suggestions for further development of the instrument:

  • Due to the negative interrater reliability and insufficient item rating scale correlation, consideration should be given to removing item 2 from the item scale.

  • For item 15, the results also showed a negative interrater reliability although the item rating scale correlation is sufficient. It is possible that the wording of the item is unclear; therefore, consideration should be given to rewording the item for clarity.

  • Item 18 also has insufficient interrater reliability. Here, the reference to “own (professional) practice” may be problematic; such a reference may be difficult for raters to comprehend and for resource developers to implement.

  • Item 24 shows negative interrater reliability. The item should possibly be removed from the item scale.

  • Item 29 has low item rating scale correlation and high difficulty. Few OER appear to have the required search function. Therefore, this aspect should possibly be omitted in a revision of the item scale.

A test-theoretical validation of the IQOER thus provided clear empirical clues for the revision and optimisation of the instrument. In the next step, the instrument will be revised according to these findings and then empirically validated again. The revision will result in an instrument with better reliability, validity, and objectivity. Overall, the optimisation of the instrument will lead to better assessment of the quality characteristics of OER.

4 Discussion

The comprehensive quality model for OER in higher education by Mayrberger et al. (2018) addresses both, the subject-specific and didactic demands of higher education institutions and the particular dynamics of the cooperative development of OER.

With the IQOER, an instrument is available for the first time that enables a reliable and valid assessment of the dimensions of this quality model. Thus, from a methodological point of view, a central prerequisite for the introduction of quality assurance measures of OER has been fulfilled.

However, for the quality assurance of OER to actually become effective, further prerequisites must be met.

The first is that quality assurance procedures must be implemented and systematically applied. Especially the providers of online repositories play a crucial role and should develop a quality assurance concept for their resources as publishers of scientific journals do. The examples in Sect. 2.1 show that such quality assurance processes are currently still very inconsistent and fragmented.

Once suitable quality assurance processes based on reliable and valid quality assessment instruments have been implemented, OER portals can ensure the quality of the resources they publish and, thus, build up a corresponding reputation among users. For users, the publication of a resource in such a portal indicates that the OER meets the relevant quality requirements.

In distributed learning ecosystems, however, the situation is more complex: Here, users find a large number of OER from different portals, making it difficult for them to assess the reputation of the source. Here, it is important that the quality of OER becomes transparent for users. Meta-repositories must make quality information visible for users for this purpose. This could be done, for example, by storing the results of quality measurements directly in metadata or by awarding quality certificates as minimum standards.

Lastly, there is a need for further research on how quality measures can be stored, shared, and communicated to users.