Methodology for Comparative Analysis of Courses

This chapter outlines the methodology for the comparative analysis of mathematics courses taught in participating Russian, Georgian, Armenian and EU universities to the students of technical and engineering programmes. It identifies a range of factors, based on which the courses have been compared, and it provides recommendations on performing the comparison. The most important part of the comparison is finding the correspondence between the actual content taught in counterpart courses of the two target universities. In order to perform this comparison in a meaningful way, the we needed a certain instrument, enabling consistent characterisation of the content of the compared courses in terms of the mathematical knowledge involved. We adopted the SEFI Framework for Mathematics Curricula in Engineering Education (SEFI, 2013) for these purposes.

Mathematical Curricula in Engineering Education [1] has been introduced to the Russian, Georgian and Armenian partners as the unified instrument to describe and compare the content of the corresponding courses.

University/Program Profile
When comparing courses, it is not enough to choose courses with similar titles. The goals and the purpose of the courses should align and the focus of the courses and their roles within the overall curricula should be comparable. Even the character of the university where a course is offered can make a difference. A classic university and a university of applied sciences can have very different perspectives on what should be the key topics within courses with the same name. In a larger university a professor can have a much richer set of resources than in a smaller one; at the same time, teaching a course to several hundreds of students puts a much bigger strain on a professor than teaching it to several dozens of them. The overall program of studies that the target course is part of is equally important for similar reasons. Therefore, the first set of criteria characterising a course profile focus on the general description of the university and the program (major) where the course is taught. These parameters include: •

Course Settings
The next set of criteria describes the context of the course including all its organisational settings and characteristics not directly related to pedagogical aspects or the content. This is the metadata of the course, which allows us to easily identify whether the two courses are comparable or not. For example, if in one university a course is taught on a MSc level and in another on a BSc level, such courses are not directly comparable, because the levels of presentation of the course material would differ much. If in one university a course's size is 3 ECTS credits and in the other 7 ECTS, such courses are not the best candidates for comparison either, because the amount of work students need to invest in these two courses will be very different even if the titles of the courses are similar. Sometimes, we had to relax some of these conditions if for particular universities best matches cannot be found. The complete list of course characteristics include: •

Teaching Aspects
In order to describe how the teacher organises the course, we identify three important criteria: use of any particular didactic approach (such as project-based learning, inquiry-based learning, blended learning, etc.), organisation of course assessment (how many tests and exams, what form they take, how they and the rest of the course activity contribute to the final grade) and the resources available to a teacher-from the help of teaching assistants to the availability of computer labs. Teaching aspects include:

Use of Technology
One of the aims of the MetaMath and MathGeAr projects is to examine and ensure the effective use of modern ICT in math education. Therefore, a dedicated group of criteria has been selected to characterise the level of application of these technologies in the target courses. There are two top-level categories of software that can be used to support math learning: the instruments that help students perform essential math activities and the tools that help them to learn mathematics. The former category includes such products as MATLAB, Maple, Mathematica, or SPSS. These are, essentially, the systems that a professional mathematician, engineer or researcher would use in everyday professional activity. Using them in a course not only helps automate certain computational tasks but also leads to mastering these tools, which is an important mathematical competency on its own. The latter systems are dedicated educational tools. They help students understand mathematical concepts and acquire general mathematical skills. In MetaMath and MathGeAr projects, we apply a particular tool like this-an intelligent education platform Math-Bridge. In both these categories, the number and diversity of available systems is very large. The focus of the criteria in this set is to detect whether any of these systems are used and to what degree, namely what the role in the course is.

Course Statistics
Another important aspect of the course is the data collected about it over the years. It shows the historic perspective and evolution of the course, and it can also provide some insights into the course's difficulty and the profile of a typical student taking a course. Although by itself this information might be not as important for course comparison, combined with other criteria it can provide important insights.

• Criterion L: Course statistics
Average number of students enrolled in the course Average percentage of students successfully finishing the course Average grade distribution Percentage of international students Overall student demographics (gender, age, nationality, scholarships, etc.) Average rating of the course by students

Course Contents
Finally, the most important criterion is the description of the learning material taught in the course. In order to describe the content of the analysed courses in a unified manner that would allow for meaningful comparison we needed a common frame of reference. As the context of mathematical education in this project is set for engineering and technical disciplines, we have decided to adopt a "Framework for Mathematics Curricula in Engineering Education" prepared by the Mathematics Working Group of the European Society for Engineering Education (SEFI) [1]. This report is written about every 10 years; and the current edition formalises the entire scope of math knowledge taught to engineering students in EU universities in terms of competences. The competencies are broken into four levels, from easier to more advanced, and allow for composite representation of any math course. As a result, every course can have its content described in terms of atomic competencies and two similar courses can easily be compared based on such descriptions.
• Criterion M: Course SEFI competency profile Outcome competencies of the course (what a student must learn in the course) Prerequisite competencies of the course (what a student must know before taking the course)

Application of the Criteria for the Course Selection and Comparison
This set of criteria should be used (1) for selecting appropriate courses for the comparison and (2) for conducting the comparison itself. During the selection process, Criteria A and B will ensure that only universities and study programs with matching profiles are selected. Criteria C, D, E and F will help to select the courses that correspond in terms of their metadata. Criteria G, H and I will help to filter out courses that utilise unconventional pedagogical approaches or differ too much in terms of assessment organisation and teaching resources available. At this point, if a pair of courses passed the screening, they can be safely compared; all criteria starting J contribute to the comparison. One needs to note that, in some cases, the strict rules of course selection might not apply, as a particular partner university sometimes presents a very unique case. In such situations, the selection rules can be relaxed.
Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.
The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.