1 Introduction

The learning management system (LMS) is a software based on a web server, cloud computing or personal local computer that manages the teaching and learning process in an academic or non-academic program without the constraint of time and place. Due to the fact that most LMSs can be accessed via an internet browser or as a user application, their accessibility is unlimited. Nowadays, there are a large number of LMS developed for the academia and industrial users [1, 2], which have leveraged new learning styles and increased the student population. For instance, the World Economic Forum (WEF) [3] documented the growth of online learners on the Coursera platform between 2016 and 2021. In 2016, there were 26 million enrollments, which increased to 189 million by 2021. This represents a growth of over 700% in just five years, on a single learning platform.

Over the past decade, Higher Education Institutions (HEIs) have significantly expanded their offerings of online programs [4]. This trend is particularly pronounced in the USA, where the entire educational system is undergoing a transformation. This transformation includes the integration of new technologies and the implementation of mechanisms that facilitate student mobility among universities through transferable college credits [5, 6]. In addition to online learning management, HEIs are revolutionizing the entire online educational system to enhance the user experience for both students and teachers [7, 8]. Given the current trends in Higher Education, this study aims to reassess the learning platforms that are currently providing their services. To achieve this, a comparative study of the most widely accessible Learning Management Systems (LMSs) data on the web are considered, reflecting the increasing use of LMSs, especially in higher education.

In this manuscript, we aim to present a brief comparison of the learning management systems, a total of 45, developed for higher education (University). This is a light report of our study, an extended technical report in Spanish is accessible in [9]. Figure 1 shows a visual summary of the educational platforms considered for the comparison. As part of our study, we examine a large number of learning management systems (LMS), which are then evaluated using a metric based on software quality and teaching-learning tools, which consists of six evaluation criteria. Then, a detailed comparison of different criteria are presented. To the best of our knowledge, this study represents the first comprehensive evaluation of a large number of Learning Management Systems. We aim to explore the relationship between relevance and impact on LMSs through two key research questions: (i) Do LMSs with a large user base achieve good scores as well? (ii) The learning management systems developed in the past decade have better features and functionalities than their previous counterparts? The findings of this research could have significant implications for educational institutions and stakeholders. If LMSs with large user bases are found to achieve good scores, it could justify investment in such systems. Conversely, if there is no correlation, institutions may reconsider their choices.

In summary, the contribution of this manuscript is as follow:

  • This manuscript presents a list of 45 LMSs developed for higher education with their basic information. Several filters were applied to the final list.

  • In order to compare LMSs, we also present a metric that combines Software Quality and Teaching-Learning tools, named SQTL to later compare with the number of user subscriptions.

  • A comprehensive quantitative comparison is given with the 45 LMSs considered in this study. In addition to SQTL, the e-learning platforms also compare the number of users, licenses, etc.

The rest of the manuscript is organized as follow: Sect. 2 presents the previous similar works shared to the community, Sect. 3 describes the procedure and materials prepared for the learning management system evaluation and comparison, the Sect. 4 presents quantitative results of each LMS considered for comparison with the respective analysis and observations, finally the work finishes with the Sect. 5.

Fig. 1
figure 1

Timeline of the LMS selected for comparison, from 1987 to 2022

Full size image

2 Related works

This section presents a literature review of the methodologies used for the LMS evaluation. Later, manuscripts that compare the LMS are tackled.

2.1 Methodologies for the LMS evaluation

From the first decade of this century, different methodologies have been proposed for the LMS evaluation. Lastly, many web-based tools have been implemented for the LMS comparison [10], although a large platforms has been considered, they do not revise LMS for higher education nor use straightforward techniques to compare upcoming e-learning platforms. In [11] on the other hand, proposes a LORIs method, it is based on technology quality and usability standards, this methodology firstly was used for the Learning Objects evaluation. Other study reported by [12], proposes a quantitative applied technique based on standard features of the learning management systems, each LMS gets an score in a range of [0–5], being 5 the best score.

Lastly, [2] presented an LMS comparison based on the software usability analysis. This evaluation method is composed by three tools: learning abilities, communication and productivity. This comparison criteria is used in the LMS for academia and enterprises.

2.2 Previous studies of the learning management systems

This section presents studies that consider LMSs for the analysis. The authors in [13] evaluated three open source platforms Moodle, ILIAS y Atutor. This study is based on user usability test, according to this technique Moodle was the best option for the users followed by Atutor. The work presented by [14] studies Moodle, Chamilo, and Google Classroom. Since Moodle is a widely used platform, the comparison is based on this software. Four indicators are considered in the comparison: (i) earnings personalization, (ii) user support, iii) tools for the student, and (iv) collaborative learning. According to the criteria proposed by [14], Moodle is the best option from the three considered for comparison. Another study is presented in [15], they studied Moodle and Blackboard. 52 features were tested, which are divided in 6 indicators: (i) pedagogical, (ii) learning environment, (iii) tools for the teacher, (iv) course design, (v) Tools for the management, and (vi) software technical specification.

Researchers in [16], performed a comparison of the following LMSs: Atutor, Claroline, Dokeos, Ilias, Moodle y Sakai. The comparison criteria developed by [17] is used, this evaluation methodology considers: (i) tools for students, (ii) supporting tools, and (iii) technical tools. Moodle and Atutor outperformed the other four platforms, since they have a better communication tools and user interface. Other comparison study analyzed six LMSs Moodle, Atutor, Blackboard, SuccessFactors, and Sumtotal [18]. The comparison is based in the literature review of each tools and features of the LMS. Those features are flexibility, user interface, accessibility, and usability. The authors conclude characterizing Moodle as a versatile, scalable, secure, and with a large set of academic tools. Lastly, [2] evaluated 35 learning management systems developed for academia and industry. The study is based on software usability analysis. It is composed by: learning abilities, communication and productivity. They concluded with the majority of the LMS considered for evaluation have similar characteristics in the three indicators.

3 Methodology

The objective of this brief comparative study is to assess the significance and influence of 45 Learning Management Systems developed for higher education. While there are only a limited number of LMSs considered for this comparison (see Sect. 2), a vast number of e-learning platforms is available on the internet, as illustrated in Fig. 1. This manuscript seeks to expand the study’s relevance by proposing new evaluation criteria and subsequently evaluating each of the 45 LMSs. Additionally, the influence of these learning platforms is examined in terms of the number of users utilizing them today. To achieve this, we have formulated the following research questions (RQ):

  • RQ1: Do LMSs with a large user base achieve good scores as well?

  • RQ2: The learning management systems developed in the past decade have better features and functionalities than their previous counterparts?

The comparison study of this manuscript needs a systematic process for the comparison (e.g., data search, data filter, comparison criteria), therefore, it follows in part the systematic review methodology proposed by [19] and PRISMA [20] (the 2021 version). This methodology is divided in three phases: (i) planning the comparison, (ii) conduct, and (iii) report the comparison results.

3.1 Comparison planning

This stage construct the best environment for a fair LMS evaluation. That is, LMS search criteria, filter procedure, and the preparation of a comparison criteria.

3.1.1 E-learning platform search criteria

The search of the LMS was conducted during May 2022 and February 2023 in Spanish and English languages, in the following search sources: Google search, Google scholar, Scopus, and the specialized web sites such as bit4learn, Capterra, e-learningindustry softwareadictive, alvarofontela, comparasoftware, edunext, getapp, adrformation, and ispring. Since many LMS are not yet studied in the academia, most of those sites are not scientific resources but technical. After considering our RQ and objectives, the keywords used in the search were: “LMS evaluation”, “Learning Management System evaluation”, “The best LMS for higher education”, “Learning Management Systems in education”, “Comparative study of LMS”, “the best LMS for the academia”.

In the searching stage, all LMSs released until 2022 were considered, January and February of 2023 did not present searching results. As the objective of the investigation requires special features of these platforms, we have outlined the selection criteria below.

3.1.2 Filter procedure

According to the search procedure detailed in the previous section, we can find around one thousand LMSs between academic and industry purpose. The study collected 108 learning management systems that offer free and straightforward access to information. These LMSs have been filtered accordingly till reach the last list for the final comparison. Below are the filter processes based on the research purpose.

  • First filter: 85 out of 108 learning management systems remain as they are not for academic purpose (Regulated educational programs).

  • Second filter: During this stage, only LMSs for high education are selected, which are 75 from 85.

  • Third filter: Due to their discontinued status or lack of official support websites, 13 LMS were avoided.

  • Forth filter: Since the evaluation criteria proposed in this manuscript is mainly based on the data collected from the official websites of each LMS, this last filter avoids all e-learning platforms that has limited information. This last filter is also prepared for the fair comparison, since lack of information does not imply that an LMS does not support or have tools that have been revised during the evaluation. Only 45 LMS passed this filter.

Before the stage four, with the purpose to consider all 75 LMSs, a survey requesting information for the comparison was sent to the official email of each LMS that had limited information. After waiting almost 2 months, 45 LMSs are considered for the further study. To answer the RQ2 is necessary the year of release of all 45 LMSs (see Table 2), therefore, if this information is not provided in the search sources detailed above, Large Language Models (LLM) like ChatGPT [21] or writeSonic [22] are used. Results from these LLMs will be highlighted in the column “Year”, in the Table 2, text in blue.

3.2 Conduct the comparison

Fig. 2
figure 2

An overview of SQTL structure

Full size image

This stage conduct the comparison of all 45 LMSs. To do so, an evaluation criteria based on software quality and tools used in online education are considered. Therefore, the name of the metric is Software Quality and Teaching-Learning tools (SQTL). For a rapid overview see Fig. 2, the text in bold are the evaluation criteria and sub-criteria of SQTL.

3.2.1 SQTL: software quality and teaching-learning tools

Table 1 Metric based on software quality and teaching-learning tools for LMS
Full size table

In accordance with Sect. 2, there are different methodologies for the LMS evaluation. Many of those methodologies are time consuming, and some LMSs are licensed, hence, individuals cannot easily access to them. Therefore, our evaluation criteria is based on the Software Quality and Teaching-Learning tools (SQTL) developed by the community from the past decades. To make a comparison, a given LMS should firstly evaluated with SQTL, which gives a respective score. To do so, the data are collected from the official websites or specialized web pages. The SQTL is constituted with 6 criteria (see Fig. 2 and Table 1) as follow: (i) Interoperability, (ii) Accessibility, (iii) Communication Tools, (iv) Productivity Tools, (v) Learning Tools, and (vi) Security and Certifications. Bellow, a deep detail about the proposed metric.

Interoperability: This refers to the ability for different systems to communicate with several different types of data in a variety of formats. Therefore, the information can be shared and accessed from many environments. The interoperability is mostly evaluated in the LMS [29, 39, 54, 63,64,65,66,67,68,69,70,71].

The maximum grade for the interoperability criteria is 1 and it should have the following file standards SCORM, AICC, LTI, xAPI/ Tin Can, QTI, XML, IMS, CMI-5, IEEE LTSC, ISO/IEC 19778-1, this correspond to the level 1 in Table 1, first row, third column; details for level 2 and level 3 are in the same row and they will reach 50% and 25% of the grades, respectively. The files formats detailed in Table 1, row Level 1 of interoperability are the most used in the online learning [25, 29, 72,73,74,75,76].

Accessibility: This criteria is defined as the capacity of the LMS to run on different hardware devices, for a vast range of users around the world [77, 78]. This accessibility has three sub-criteria: LMS based on web navigator, compatibility with different hardware (e.g., PC, tablet and smartphones), and support 4 languages or 3 most spoken language in the world. those four language are English, Chinese, Hindi and Spanish [79]. If the LMS is based on the web navigator; it is compatible with PC, tablet, and smartphone; and support the language mentioned previously, it reaches 100% of the score for the accessibility criteria, see details in row two of Table 1.

Productivity tools: Studies from [39] and [80], suggest, this tools are essentials for the LMS. According to the those authors, a good LMS should manage courses, documents, users, and grades. Therefore, the row three in Table 1 resume the grades of the three levels for productivity tools. For example, an LMS with a grade of 0.5 points (just the 25%) is the level 3.

Communication tools: These tools facilitate the teachers and students interactions in an online educational platform. Therefore, a vary of such tools are needed [16, 18, 20, 80,81,82,83,84,85]. It has two sub-criteria, synchronous and asynchronous communication tools. According to the authors mentioned previously, each sub-criteria has the respective important tools as shown in Table 1 (5th row). To reaches the best score (2 points) in this row, each sub-criteria has to reaches 50%. In level 1, for example, 4–5 Synchronous tools should have a leaning management system. To give another example, the column of the level 3 of this criteria correspond to a 25% of the grade (0.5 points), see Table 1 4th row, video-conference correspond to synchronous tools; forum and email to asynchronous.

Learning tools: As many studies suggest, this criteria is one of the most important tools that any LMS should be equipped for educational software [39, 80, 86,87,88,89,90]. There are a different kind of modules to make activities and contents that facilitate the learning of the students, hence, we have divided this part in three sub-criteria (i) Tools for understanding and internalizing contents, (ii) tools to promote collaborative and individual learning, and (iii) tools for the contents management and distribution.

The maximum grade for this criteria is 2 points that corresponds to level 1 (100% of the grade). The following sub-criteria are incorporated into this criteria, which includes the following tools:

  • Understanding and internalizing contents: Survey, Choice/queries, search into the course, video service/virtual blackboard, document sharing, Calendar, support/help, diary, and predefined surveys. Having 7–9 of these tools correspond to the 25% of the grade (0.5 points).

  • Promote collaborative and individual learning: Forum, Wiki, assignments, chat, gamification, workshop, quiz, blogs, lesson, glossary, and groups. If an arbitrary LMS considered for evaluation has 9–11 tools, 50% of the level one grade is given to this sub-criteria. Since those tools mostly contribute to the educational process [62].

  • Contents management and distribution: Archive/Documents, URL, external tools, folder and SCORM. An LMS having 4–5 tools, receives 25% of the grade.

This three sub-criteria sum 100%, 2 points. The tools ordered in the items above are sorted according to its importance [14, 51, 82, 84, 91,92,93]. To review a complete scale of quantification corresponding to the learning tools criteria see Table 1, row 5th.

Security and certifications:This criteria tests the reliability and security in a software. That is, it is important the LMS has user authentication, access verification, integrity control, detect non authorized users, as well as different security certifications [18, 62, 80, 94,95,96,97,98,99]. To quantitatively evaluate this criteria we have divided it in two sub-criteria, security standards and security certifications, see Table 1, last row.

The level 1 of this criteria gets 2 points, 50% from security standards (first sub-criteria) and the other 50% for security certifications. In the first sub-criteria, the LMS, to reach the respective grade, should have at least TLS/SSL, SSO, GDPR and HTTPS standards. And in the security certifications, the LMS gets 50% of the grade if at least has 5 to 7 of the following certifications ISO/IEC 27001, SOC 2, DIACAP, FedRAMP, HIPAA, ISO 9001, ISO 27018. Following the same procedure, the grades for the level 2 and 3 get \(2\times 0.5\) and \(2\times 0.25\) points, respectively.

4 Results

In this section we present a quantitative comparison, mainly using the SQTL evaluation methodology presented in Sect. 3.2. Later, we answer the research questions formulated in Sect. 3.

Table 2 General information of the LMS considered for comparison and the SQTL results
Full size table

Table 3 presents general information of LMS considered for comparison, there are 45. The first column (No.) correspond to the indexing number. The second column corresponds to the learning management system name. Third column contains the objective user for whom the software was created; the segment of people considered are Enterprise (EN) and Academic (AC)—this set has Primary Education (PE), Secondary Education (SE), and High Education (HE). LMS realised countries are listed in the column Country. The column Year is the year of the LMS realise. The Users has the total number of users in millions (M). The column License has commercial when the LMS is payed, Free stands when the LMS is Freeware or open-source, if Both is in the cell, the LMS has two type of licenses. The last column SQTL stands for the quantitative result of each LMS evaluated with our proposed SQTL. For example, the last row shows Academy LMS developed for Enterprises and Academia, released in USA in 2022, it just has 0.0008 million (near to 1K) users, it only has commercial license and the SQTL score is 4.75 points. Every row of Table 3 has similar description but each row correspond to the LMS information. The table is stored according to the year of release.

According to Table 3, the majority of the LMSs are for academic and industry users. Most platforms are released in America and Europe. This study tackles software released from 1987 until 2022, more than half of those LMSs have been realised from 2010 and most of them are licensed as Commercial. The SQTL score is in the last column, as in most of the educational system the grade range in SQTL is [0–10], being the range of 9–10 the best score, 7–8 a very good grade, and 5–6 an acceptable LMS score. The \(89\%\) of the LMSs considered for comparison reach acceptable, very good, and best scores.

Table 3 SQTL detailed result
Full size table

A detailed revision of SQTL results can be seen in Table 3. As presented in Sect. 3.2, SQTL is composed by 6 criteria, each result can be seen in Table 3. For example, in the row of Moodle summing all 6 results, [Interoperability \(+\) Accessibility \(+\) Productivity tools \(+\) Communication tools \(+\) Learning tools \(+\) Security and certifications] it reaches 9.25 points, see last column of this table. Overall, we can see that most of the LMS fulfil the requirement of Accessibility and Productively tools criteria.

Fig. 3
figure 3

Top 10 learning management systems according to the number of users. Additionally, SQTL results are provided

Full size image

From the Table 2, we present the top 5 LMS with most users in the Fig. 3. Additionally, SQTLs’ score of each LMS is also presented in the figure. As illustrated in Fig. 3, Moodle is the top 1 according to the number of users, at the same time, it reaches the best result range of SQTL, followed by Blackboard LMS. All LMS in the top 10 reach greater than 5 points, which suggest, most of the users use LMSs with good tools and features.

Fig. 4
figure 4

Top 5 free and commercial LMSs according to the SQTL criteria

Full size image

The Fig. 4 presents top 5 best scores in SQTL according to the type of license, bars in red correspond to commercial license and bars in blue to free or open source platforms. As the figure shows, the best result correspond to Paradiso LMS, which scored 9.5 points, this platform is followed by Moodle (9.25). Paradiso is a young LMS that only has around 1 M users, however, Moodle has over 250 M users. In comparison with the top five commercially licensed e-learning platforms, the five free platforms achieve better SQTL results, only Moodle loses with a small difference of 0.25 points.

4.1 Do LMSs with a large user base achieve the best scores as well?

Firstly, to respond this research question, we tackle Fig. 3 and Table 2. According to that figure just Moodle reaches the best result range (9–10) and also has more users than other LMS. Blackboard is the second best LMS in terms of Users and SQTL score. Secondly, in the Table 2, we can see that the best top five LMSs, according to SQTL are: Paradiso, Moodle, Chamilo, Canvas, ProProfs, and Sakai, their scores are 9.5, 9.25, 8.75, 8.38, 8.0, and 8.0, respectively. However, if we compare the Moodle users with the combined users of the other five LMSs, they are just 30% of Moodle. According to these observations, the answer is No, one of the reason may be the lack of access to information about the learning management systems, but a deep study is requested for a better understanding of this behavior.

4.2 The learning management systems developed in the past decade have better features and functionalities than their previous counterparts?

To address this question, we tackle the SQTL results from the Table 2. To compare the 45 LMSs, we divided them into two groups according to their year of release. A first group includes the LMSs released before 2010, while the second group includes the rest. The LMSs released in the years of 1987 to 2009 get 6.91 average score. On the other hand, the last learning management systems reaches 6.40 average score. Although the SQTL best score is Paradiso LMS (released in 2013), the learning management systems launched before 2010 overtake the average score of recent LMS by 0.51 points. That is, the e-learning platform developed before 2010 have slightly better features and functionalities than the LMS released after 2010.

4.3 Discussion and limitation

As our results in Table 3 are based in the information provided by the official web site and technical and specialized web pages in LMS, the results may avoid some information that, actually, the LMS has in its current version but it was not mentioned in those websites neither in this manuscript. However, it should be mentioned that most of the information documented in the Table 2 and 3 were double checked by three authors of this manuscript. Every data collection and productions documented in this work is only for research purpose. This work is performed since, as tackled in Sect. 2, previous comparison study selects from 2 to 10 LMS [18, 32, 100]. Our work may shares similarities with the comparative study of 36 LMSs discussed in [2]. However, our study distinguishes itself in several key aspects that are not aligned with the objectives outlined in this manuscript. Firstly, [2] examines LMSs using scientific works published between 2004 and 2019. Secondly, the LMS selection is performed based on keywords search on Google and Bing search engines. Our study on the other hand is different in the following terms: (i) the comparison study of LMS is for higher education, (ii) we expand upon the features evaluated in [2] by including assessment of interoperability, accessibility, and software quality, which is termed as SQTL, (iii) additional to SQTL evaluation, the study presents more information such as year of release, number of users to name a few, iv finally, our LMS selection criteria involved a comprehensive approach, including searches conducted in both English and Spanish, across various search platforms and specialized websites. Due to the importance of presenting the relationship between popularity and technical score, the number of users of the LMS is considered in the comparison.

Several LMSs have been awarded and highlighted in many academic and social events. However, their popularity is limited. This is likely due to the lack of open access to research results, both for the academic and industrial community. Therefore, a comparative study of the Learning Management Systems with the best benefits for university education has been carried out. Since the analysis of each LMS is time consuming and the access to information, due to many situations, is very limited, we present SQTL to study the 45 e-learning platforms that reduce those disadvantage in the LMS analysis. The SQTL that evaluate the LMS for Higher Education is based on the criteria constructed with the academic works reported thorough the last two decades. There is no use of user-centered criteria in the study of this work.

5 Conclusions

A brief comparative study of a large learning management systems has been studied in the manuscript. The software quality and teaching-learning tools based metric (SQTL) is proposed based on manuscript published from the last two decades. This study evaluates 45 platforms using SQTL and presents them for comparison. It can serve as a valuable reference for Higher Education Institutions that offer online programs and use eLearning platforms. The findings and scores presented in this study serve as valuable resources for future research and the enhancement of specific LMS. According to our evaluation criteria, Paradiso and Moodle emerged as the top-performing LMSs, with only a slight difference of 0.25 points, despite their differing commercial and free licensing structures. The information regarding these LMSs was current as of February 2023.