Changes in Students’ Mathematical Competencies at the Beginning of Higher Education Within the Last Decade at a German University

Abstract

Mathematics plays a significant role in many study programs. However, several studies show deficiencies and a decline in beginning undergraduates’ skills in mathematics in many content domains. However, it remains unclear whether they have improved in so-called process competencies like modeling, mathematical reasoning, or using different representations instead, because there has been a shift towards the acquisition of such in many recent curricula. We investigated this issue at a university in Germany based on data from a (non-standardized) mathematics entry test taken by 3076 economics students divided into different cohorts from 2012 to 2019. Using regression analyses, we found that, on the one hand, students’ ability to carry out symbolic calculations decreased. On the other hand, their performance increased in some test questions focusing on other process competencies like reasoning, mathematizing, or using different representations, which have become common tasks at school due to a stronger emphasis on these process competencies after a curriculum reform. Our data indicate that this reform might have had the desired effect.

Introduction

Mathematics is central in many study programs like engineering, the natural sciences, and economics. Therefore, beginning undergraduates should have a solid qualification in mathematics at the end of secondary school. However, there is some evidence that this is often not the case (Bingolbali & Monaghan, 2008; Cook & Fukawa-Connelly, 2016; Durandt et al., 2022; Galbraith & Haines, 2000; Jankvist & Niss, 2020; Kempen & Biehler, 2019; Kendal & Stacey, 2003; Lithner, 2000; Moore, 1994; Mullis et al., 1998; Trigueros & Ursini, 2003). What has been investigated much less often are changes in students’ mathematical qualifications at the entry to tertiary education. This issue is crucial for an evaluation of school curriculum reforms, which have been initiated as a response to students’ poor performance in mathematics in many countries, for instance, the USA, Great Britain, Ireland, and Germany (Hodds et al., 2020; KMK [Conference of the ministers of education of Germany], 2003, 2012; National Council of Teachers of Mathematics, 2000; Treacy & Faulkner, 2015). These reforms aimed at shifting the focus away from mathematical procedures students should be able to carry out correctly to competencies they should possess at the end of secondary education (Boesen et al., 2018). Besides content-specific skills, these competencies especially cover general mathematical processes like modeling, mathematical reasoning, problem-solving, using different representations, or communication (see, e.g., KMK [Conference of the ministers of education of Germany], 2012; National Council of Teachers of Mathematics, 2000).

There only exist some empirical studies that explicitly focused on changes in students’ mathematical skills at the entry into tertiary education (Faulkner et al., 2010; Gill et al., 2010; Hodds et al., 2020; Hunt & Lawson, 1996; Lawson, 1997, 2003; Treacy & Faulkner, 2015; Treacy et al., 2016). Many report a decline in students’ overall or content-specific mathematical skills. However, the test questions used in these studies primarily focused on calculation procedures. So maybe, students have improved in mathematical processes like mathematical reasoning or modeling instead? We did not find a study investigating this issue thus far, although this is particularly important considering the school curriculum reforms mentioned above. The study we present here attempts to fill this gap.

Theoretical Background

Literature Review and Embedding of the Research

Students’ Skills in Mathematics at the Beginning of Tertiary Education

Several studies point out that many students have gaps in mathematical knowledge and skills at the end of secondary school or when entering tertiary education.

Firstly, many studies show problems in students’ mastery of certain mathematical content. These often indicate that although students can carry out mathematical procedures correctly, they do not understand the underlying concepts. This particularly applies to fundamental concepts of calculus like the derivative (Bingolbali & Monaghan, 2008; Mullis et al., 1998), but also basic algebraic concepts such as the concept of variable (Trigueros & Ursini, 2003) or basic concepts of statistics like mean, median, or standard deviation (Cook & Fukawa-Connelly, 2016).

Furthermore, several studies indicate that many students have difficulties in mathematical processes like modeling, mathematical reasoning, or using different representations at the end of secondary education. First, some studies indicate deficiencies in modeling (Durandt et al., 2022; Ikeda & Stephens, 1998; Jankvist & Niss, 2020). Jankvist and Niss (2020), for example, investigated Danish students’ difficulties with modeling. They administered six modeling tasks to 315 students at upper secondary schools. Many of these failed to solve the problems administered, for example, because they could not adequately mathematize the situations given in the tasks or could not work with a model set up if numbers needed for this work were not given precisely. Altogether, the students had issues in many different stages of the modeling process.

Another mathematical process many students have not mastered at the end of secondary school is mathematical reasoning (Kempen & Biehler, 2019; Lithner, 2000; Moore, 1994). Kempen and Biehler (2019), for instance, found that many students of a first-semester course considered an empirical verification with examples as a suitable method for mathematical reasoning. Hence, they had an insufficient idea about what kinds of arguments are acceptable for mathematical reasoning. Furthermore, Lithner (2000) found in a study on students’ reasoning when solving mathematical problems that these based their reasoning rather on remembered procedures than on the mathematical components involved in the problems.

Concerning the use of different representations of mathematical objects, studies indicate that beginning undergraduates particularly have problems with switching between different representations, for example, between different representations of the function concept (Galbraith & Haines, 2000). Also, correctly interpreting data from diagrams is a hurdle for many students at the end of secondary school, as it is, for instance, suggested by the published items of the TIMMS study (Mullis et al., 1998).

Overall, the literature suggests that although many beginning undergraduates can carry out routine procedures correctly, they often have an insufficient understanding of the underlying mathematical concepts. Furthermore, they have deficiencies in important mathematical processes like modeling, mathematical reasoning, or using different representations of mathematical objects or phenomena.

Changes in Students’ Mathematical Skills at the Beginning of Tertiary Education

Fewer studies focused on changes in students’ mathematical skills at the beginning of tertiary education. We essentially found two essential chains of studies: one conducted in England at Coventry University (Hodds et al., 2020; Hunt & Lawson, 1996; Lawson, 1997, 2003) and one in Ireland at the University of Limerick (Faulkner et al., 2010; Gill et al., 2010; Treacy & Faulkner, 2015; Treacy et al., 2016).

The first of these studies was conducted at Coventry University in England between 1991 and 1995 and focused on changes in the mathematical skills of low-performing students (Hunt & Lawson, 1996). It was based on a mathematics diagnostic test developed by the BP Mathematics Centre—a support facility for undergraduate engineers at this university. The test has been administered to science and engineering students at the beginning of their study program since 1991. The test consisted of 50 multiple-choice questions covering seven topics: basic arithmetic, basic algebra, lines and curves, triangles, further algebra, trigonometric functions, and calculus. Hunt and Lawson (1996) compared the test results of students with the grades D (one grade above minimum standard), E (minimum standard), and N (nearly passed) in their mathematics A-level from the years 1991, 1993, and 1995. They found that the average score decreased between 1991 and 1995 for all topics in all groups (with one exception in the topic “triangles”). Many of the score differences between the different cohorts were significant. Hence, the data suggest that there was a decline in students’ mathematical skills in many content domains. Similar results are also documented in Lawson (1997) and Lawson (2003), comparing the results of the same test between 1991 and 1997 as well as between 1991 and 2003.

The last study conducted at Coventry University was published recently (Hodds et al., 2020). This study focused on changes in students’ performance in the test mentioned above from 2001 to 2017. Hodds et al. primarily investigated whether changes in the A-level mathematics curriculum since 2001 (e.g. the introduction of a modular system) might have affected students’ performance. They then found that the number of students who had taken an A-level in mathematics had grown in the period observed and that the average score in the diagnostic test increased from 2001 to 2017 among these students, also for low-achieving students with the grades D or E. However, their scores remained lower than those in 1991, and the gap to students with high grades in their A-level has increased since 2001. Furthermore, the heterogeneity has grown. Nevertheless, the study shows an overall improvement in students’ mathematics entry skills from 2001 to 2017.

The second chain of studies investigating changes in students’ mathematical skills at their entry to higher education was conducted at the University of Limerick in Ireland (Faulkner et al., 2010; Gill et al., 2010; Treacy & Faulkner, 2015; Treacy et al., 2016). These studies were based on a mathematics diagnostic entry test designed by a mathematics education professor at this university in 1997 (Faulkner et al., 2010). It consisted of 40 open questions (38 calculation questions and two questions asking to sketch the graphs of the functions $y=3x+2$ and $y={x}^{2}+2$). Since 1998, all first-year Technological and Science students have been required to take this test in their first lecture without warning. Faulkner et al. (2010) found that the average score decreased significantly between 1998 and 2008. However, in contrast to the findings by Hunt and Lawson (1996) or Lawson (2003), the average performance remained the same for students with the same grade in mathematics on their school leaving certificate.

The other two studies at the University of Limerick cover later years (Treacy & Faulkner, 2015; Treacy et al., 2016). Treacy and Faulkner (2015) investigated the overall performance from 2003 to 2013. Again, the average score decreased significantly—now even among students with the same grade on their school leaving certificate. Furthermore, Treacy and Faulkner (2015) recognized that the number of students “at risk” (the ones who mostly answered 18 of 40 questions correctly) increased substantially. Treacy et al. (2016) then focused on students’ performance in the different content sections of the test—arithmetic, algebra, geometry, calculus, and modeling (only one question)—from 2008 to 2014. They found that the average score of students with grades C and D decreased significantly in all topics except for modeling in this period. Students with grade B even improved in modeling significantly. Since a curriculum reform in Ireland in 2014 emphasized using mathematics in contexts, these data did not only show that the mathematical skills of low-performing students declined, but also indicate that the curriculum reform might have had a positive effect on students’ modeling skills. However, these conclusions remain speculative since the test had only one modeling question, and the data contained only one cohort who had completed school with the new curriculum.

Overall, the studies described above, except Hodds et al. (2020), indicate that students’ mathematical skills at entry to university have been declining within the last decades—overall and in different content domains. However, the question arises whether students have improved in mathematical processes like modeling or mathematical reasoning, particularly because recent curricula often foster these. The studies presented above cannot answer this question because the items in the tests used were almost solely calculation questions (Faulkner et al., 2010; Hunt & Lawson, 1996). Therefore, we collected data from a university mathematics entry test between 2012 and 2019 whose items also addressed mathematical processes like reasoning, mathematizing, or using different representations. We analyzed the data from a competency perspective to find a first answer to the research question:

To what extent have students’ mathematical competencies at entry to tertiary education changed within the last decade?

Theoretical Framework of Our Study

Traditionally, curricula have been based on lists of topics students should master—for example in Barry and Steele (1993) for engineering students. This mastery especially covered the ability to carry out procedures related to these topics and to understand the underlying mathematical concepts, including the relations between these. Hiebert and Lefevre (1986) call these two issues procedural and conceptual knowledge. However, Freudenthal (1973) already pointed out that mathematics is not just a body of knowledge but also an activity, and that the activity of creating mathematical knowledge should also be part of mathematics education (pp. 114). Seizing this idea, Törner and Grigutsch (1994), who researched students’ views on mathematics, distinguished between two different “philosophies of mathematics”: mathematics as a static system and mathematics as a process. In the former, mathematics is a (finalized) system of axioms, definitions, and theorems providing procedures for dealing with specific tasks. In mathematics as a process, the focus does not lie on “applying” definitions or theorems but on (re)inventing these based on questions and problems. Törner and Grigutsch (1994) especially highlighted the following activities as fundamental in mathematics as a process: exploring, modeling, problem-solving, conjecturing, and reasoning.

One framework that used the abovementioned ideas when describing what it might mean to have mastered mathematics was developed in the Danish KOM project (Niss, 2003; Niss & Højgaard, 2019). In this project, a committee of mathematicians, mathematics teachers, and researchers aimed to answer this question (due to practical educational problems occurring in Denmark at that time). Their work provided a framework that gave a theoretical description of what it might mean to have mastered mathematics—independently from the specific mathematical subject matter and educational levels (Niss & Højgaard, 2019).

They based their framework on the notion of competence, which they originally defined as follows (Niss, 2003, p. 6):

To possess a competence (to be competent) in some domain of personal, professional or social life is to master (to a fair degree, modulo the conditions and circumstances) essential aspects of life in that domain.

Since competence in a field cannot be realized independently from human beings and is more like a property of a human being that can be put to use in certain situations, Niss and Højgaard (2019) later adapted their definition of competence as follows (p. 12):

Competence is someone’s insightful readiness to act appropriately in response to the challenges of given situations.

As the phrase “to act” indicates, this definition is oriented towards actions—including mental actions. But the individuum can also consciously decide to refrain from any action, which Niss and Højgaard highlighted with the word “readiness.” Second, they emphasized that readiness refers to the individual’s cognitive prerequisites only, and not to its dispositional, affective, or volitional traits. This is, of course, questionable, and several other researchers and institutions view the latter also as an essential part of mathematical competence (Niss et al., 2016). However, since we especially wanted to investigate whether students improved in mathematical processes like reasoning, modeling, or using different representations that were all included in the framework by Niss and Højgaard (2019) explicitly, we nevertheless consider their focus on cognitive aspects of competence as suitable for our research.

Following their general definition of competence, Niss and Højgaard (2019) then defined mathematical competence as follows (p. 12):

Mathematical competence is someone’s insightful readiness to act appropriately in response to all kinds of mathematical challenges pertaining to given situations.

They specified that “act appropriately” particularly involves several components: knowing, understanding, doing, using, and judging mathematics (p. 12). They furthermore emphasized that factual knowledge and technical skills are an essential part of mathematical competence, but that mathematical competence cannot be reduced to these issues.

The committee of the KOM project then identified eight mathematical competencies that they considered “major constituents” of mathematical competence. In the following, we will summarize these shortly; due to limited space, we do not present precise definitions from Niss and Højgaard (2019).

• 1. Mathematical thinking competency: It involves being able to relate to and to pose typical questions that are characteristic of mathematics (examples: Does there exist? How many? Does the inverse implication hold as well? If an object has property A, does it then need to have property B as well?), and to relate to the answers one might expect to such questions. It further involves distinguishing between different types of mathematical statements (definitions, if–then claims, existence claims, conjectures, and so on) and reflecting on the role of the quantifiers in these statements. Finally, it involves reflecting on the scope of mathematical concepts or terms and reflecting on and proposing abstractions of mathematical concepts.

• 2. Mathematical problem handling competency: It involves posing and solving different kinds of problems for which a standard procedure is unknown, and analyzing solutions to such problems.

• 3. Mathematical modeling competency: It involves using mathematics to answer extra-mathematical questions, particularly constructing mathematical models and critically analyzing and evaluating mathematical models proposed.

• 4. Mathematical reasoning competency: It involves analyzing and producing mathematical arguments (also at an informal level) to justify mathematical claims.

• 5. Mathematical representation competency: It consists of the ability to interpret and move between different representations (e.g. verbal, material, symbolic, graphical) of mathematical objects, phenomena, relationships, and processes.

• 6. Mathematical symbols and formalism competency: It involves successfully handling mathematical symbols and expressions (including the construction and decoding of such).

• 7. Mathematical communication competency: It includes communicating mathematics in different styles, registers, and at different levels.

• 8. Mathematical aids and tools competency: It involves dealing with material aids and tools for mathematical activity. This covers physical instruments as well as technical tools.

These competencies are meant to be distinct, i.e. each competency has a well-defined identity but is not disjoint (Niss & Højgaard, 2019). They form the theoretical basis for our research, as we especially aimed at finding out whether students have improved in these mathematical process competencies within the last decade.

Methodology

To investigate possible changes in students’ mathematical competencies at the beginning of tertiary education within the last decade, we used data from a mathematics entry test conducted at the beginning of each winter semester from 2012 until 2019 at a midsized university in Germany. We analyzed this data from the competency perspective just described—also to evaluate a possible effect of a curriculum reform in Germany that shifted the focus away from mathematical procedures students should be able to carry out towards the acquisition of the competencies mentioned above (KMK [Conference of the ministers of education of Germany], 2003, 2012). For this,

• 1. We developed a classification scheme for the test tasks based on the theoretical competency framework mentioned above,

• 2. Classified the tasks according to the competencies addressed in them,

• 3. Clustered the tasks according to the core competency addressed, and

• 4. Carried out regression analyses which included the time as an independent variable.

Participants

We gathered data from 3254 beginning undergraduates in economics at a midsized university in Germany from 2012 to 2019 who had not taken the introductory mathematics course of the economics study program yet. The sample covered about 90% of all students enrolled in the program in that period. Of these 3254 students, only 178 needed to be excluded due to missing data. The remaining 3076 students represented a relatively broad spectrum of students that enter university in Germany, as only 57% of these had acquired the “Abitur”—the German general university entrance certificate. The remaining proportion has been admitted to the study program with the so-called Fachhochschulreife (FOS) —a certificate that usually only admits to universities of applied sciences but may also admit students to some programs at university. Such “FOS-students” are required to take fewer courses of standard school subjects like mathematics in grades 11 and 12 (but have other subjects like economics instead), and the corresponding schools have a more application-oriented focus.

As already said, the curriculum had been reformed within the last decade towards the acquisition of the aforementioned mathematical competencies. Since in Germany, it is not the central government but the governments of different regions (“Bundesländer”) who have the responsibility for the school curricula, there is no uniquely determined point at which all of our participants have been exposed to a new curriculum. Nevertheless, one can assume that all students in our later cohorts have been taught according to such a competency-focused curriculum.

Instruments and Data Collection

We investigated students’ competency changes using a paper and pencil test. The test was designed by the teacher of the mathematics course for economics students at the university of our participants and by a mathematics education researcher from the same university. It has remained the same since 2012 and consisted of 30 items that covered the following content areas: arithmetic, algebra, functions, and calculus. The tasks mainly focused on skills the teacher considered as essential for success in his course “Mathematics for economics students”, but also included tasks addressing mathematical processes like mathematizing, reasoning, or using different representations of mathematical objects (Laging, 2021). Four sample tasks are shown in Fig. 1. Task 11, for instance, requires a mathematization of the situation, task 25 requires switching between different representations, and task 30 requires mathematical reasoning. Furthermore, the last two tasks in Fig. 1 can be considered as problems for which no standard procedure has been taught. Nevertheless, all tasks are solvable based on the knowledge covered at school.

Fig. 1

Sample tasks of the mathematics entry test

Furthermore, we gathered information on our participants’ socio-demographic backgrounds. In a questionnaire accompanying the test, we asked them about several socio-demographic characteristics that had been shown to impact students’ performance in our test (Laging & Voßkamp, 2017). These were: gender, optional participation in a 2-week mathematics preparatory course before the semester offered by the staff of the introductory mathematics course of the economics study program, the years between the high school degree and the start of the study, and the type of entrance certificate that admitted our participants to the study program. Especially the entrance certificate type and participation in the preparatory course have been identified as strong predictors of test performance in previous research (Büchele, 2020; Laging & Voßkamp, 2017). In addition, we asked for the year of study because some of our participants took the introductory mathematics course of the economics study program not in the first but a higher semester.

For data collection, our participants have been required to take the test just described as a mathematics entry test (paper and pencil), and to complete the questionnaire at the beginning of their introductory mathematics course “Mathematics for economics students” each winter semester since 2012. They had about 90 min to solve the test and complete the questionnaire. Data were raised anonymously and within the university's ethical standards.

Data Analysis

Classification of the Tasks According to the Competencies Addressed

Development of a Competency Classification Scheme

We developed a classification scheme based on our theoretical framework to analyze the test tasks according to the competencies addressed in them. Since the tasks often addressed more than one competency, we developed a level-categorization scheme with four levels for each competency (from 0 to 3). Similar to PISA (Organization for Economic Cooperation and Development [OECD], 2018), these levels were meant to indicate the demand of a task. But since we wanted to classify the demand in relation to the different competencies of our framework, we did not use a general construct of cognitive demand as in PISA that describes a task’s overall demand (OECD, 2018). Instead, we decided to develop a level scheme that should show for each competency the extent to which it is required to master the task—comparable to what Turner et al. (2015) did for the competency framework used in PISA 2012.

The development of the classification scheme covered several steps to ensure that the classification would be as objective as possible:

• 1. Initial formulation of the competency levels and the corresponding coding rules based on the precise definitions of the different competencies by Niss and Højgaard (2019),

• 2. Initial classification of the tasks by the two authors and a mathematics education student,

• 3. Discussion of difficulties experienced during this initial classification (e.g. if level descriptions were not precise enough to allow a distinct classification) and further specification of the coding rules,

• 4. Completion of the initial classification of the tasks,

• 5. Discussion about systematic code differences and final adaption of the coding rules.

The resulting final classification scheme and final coding rules are shown in Fig. 2 (we omit the description for the mathematical aids and tool competency because no aids have been allowed in the test).

Fig. 2

Classification scheme for the competencies addressed in the tasks

This scheme needs four specifying remarks regarding the coding, which resulted from discussions about problems occurring during the initial classification of the tasks:

• Like Boesen et al. (2018), we coded the competencies addressed in the tasks based on what was required in the intended solutions (we also discussed the intended solutions with the instructor of our participants’ introductory mathematics course).

• Concerning the mathematical thinking competency: The competency level for each task was decided according to whether the students were required to pose, answer, or reflect on the “generic questions in mathematics” mentioned in the competency definition, like “How many?” or “Does there exist?” by themselves. The level was assigned to 0 if an answer to such a question was already given in the task as in “Show that the two graphs have no intersection point”.

• Concerning the mathematical problem-handling competency: According to Niss and Højgaard (2019), this competency only refers to inner-mathematical situations. Therefore, we decided the level of this competency for word problems requiring a mathematization by considering the inner-mathematical steps of the intended solution only.

• Concerning the mathematical representation competency: Unlike in the definition by Niss and Højgaard (2019), we only regarded the usage of representations aside from mathematical symbols as part of this competency so that it discriminates from the mathematical symbols and formalism competency.

Final Coding and Reliability Analysis

After we had developed the final coding scheme, we (the two authors) waited some time and then classified the tasks anew with the final coding scheme. We then checked the interrater reliability with Cohen’s Kappa (Table 1). The values are all above 0.6 and can therefore be considered good (Landis & Koch, 1977).

Table 1 Interrater reliability of the coding

Assignment of a Core Competency to Each Task of the Tests

We then assigned a “core competency” to each task by choosing the competency with the highest mean of the two codes. If this mean was equal for two competencies, we chose the one that we considered crucial to master the largest hurdle in the task. Task 11 from Fig. 1, for instance, is such a “border example.” It requires setting up and working with linear equations (level 1 of the modeling and level 1 of the symbol and formalism competency, see Fig. 2). However, since handling linear equations is practiced extensively at school while setting up such is not, we judged the mathematizing step as more critical, and finally assigned the task to the modeling competency. Another example that had equal values for two competencies was task 25 from Fig. 1. It addresses problem-solving and switching representations. Although this is not a standard task at school and hence also addresses problem-solving, the idea of using the geometric interpretation of the derivative in such a graphical task involving the derivative is rather apparent. The crucial hurdle is then to translate the symbolic information about the derivative given onto the graphic level correctly. We therefore assigned this task to the representation competency.

The final core competencies and the corresponding levels are shown in Table 2—values with a decimal result from different ratings between the two authors.

Table 2 Core competencies of the different tasks and corresponding levels

This table especially shows two crucial issues. First, tasks focusing on the mathematical symbols and formalism competency were overrepresented in the test, because the test mainly focused on skills the course teacher considered as essential for success in his course. This is a significant limitation of the test instrument. But Table 2 shows as well that the test also contained several tasks with modeling as the core competency and several tasks focusing on the use of mathematical representations aside from mathematical symbols.

Aggregation of the Tasks in Competency Clusters

Finally, we clustered the items with the same core competency. Since the test only contained few items with the core competencies “Mathematical problem-handling,” “Mathematical reasoning,” and “Mathematical communication,” we did not create separate clusters for these competencies. Instead, since the tasks with these core competencies addressed at least three competencies with a level higher than 0 and were the only ones that did so, we clustered these four tasks (6, 20, 23, 30) together as “tasks addressing multiple process competencies” for our further analysis.

Analysis of the socio-demographic characteristics

Coding of the Socio-demographic Variables

The coding scheme for the socio-demographic data collected is shown in Table 3.

Table 3 Coding of the socio-demographic data gathered

Analysis of the Socio-demographic Characteristics

Table 4 gives an overview of the socio-demographic characteristics in our sample—overall and in the different cohorts. These variables had been identified to affect students’ test performance in previous research (see section “Instrument and Data Collection”). Since they vary between the different cohorts and do not coincide with the values of all students in Germany, it was necessary to control them in our subsequent trend analysis so that the results would be less biased.

Table 4 Means/shares of the socio-demographic variables over the cohorts

Statistical Approach for Investigating Trends

To investigate whether students’ competencies have changed within the period observed, we carried out a regression analysis with the total test score as the dependent variable and the time and the socio-demographic control variables  ${B}_{1}$ to ${B}_{7}$ from Table 3 as independent variables. This led to the following regression model in Stata:

$$Y=constant+ \mathrm{\alpha }*time +\sum\nolimits_{j=1}^{7}{\upbeta }_{j}*{B}_{j}+\upepsilon$$

(1)

$Y$ is the score achieved, $\alpha$ is the estimator for the trend, ${\beta }_{1}$ to ${\beta }_{7}$ are the estimators for the given control variables ${B}_{1}$ to ${B}_{7}$, and $\epsilon$ is an error term. To counter heteroscedasticity, we estimated a model with robust standard errors. The variable “time” was coded as 1 to 8 (1 for the 2012 cohort and 8 for the 2019 cohort). We determined several trends this way:

• 1. The trend in students’ overall test performance,

• 2. The trends in students’ mathematical competencies on the level of the above-defined competency clusters,

• 3. The trends in students’ performance on the level of individual tasks

In a final step, we also included interaction effects of the grade in mathematics and the higher education entrance certificate type into the regression to control for skill differences over time.

Results

In this section, we will present the results of our trend analyses, which are structured as follows:

• 1. The trends in the students’ overall test performance and the different competency clusters

• 2. Interaction effects of grade and type of entrance certificate with these trends

• 3. The trends in students’ performance in the individual tasks

Trends in the Overall Test Performance and the Different Competency Clusters

Table 5 provides the trend analysis results for the overall test performance and the competency clusters defined above.

Table 5 Results of the trend analysis concerning the overall test and the different competency clusters, **p < 0.05

The trend analysis reveals a slightly negative but insignificant trend for the overall test performance. The numbers in Table 5 can be interpreted as follows. The trend coefficient $\alpha$ means that the students performed on average 0.031 points worse each year. Hence, the students in 2019 performed on average 0.248 points lower than those in 2012. The fraction $\frac{\mathrm{Trend}}{\mathrm{Mean }2012}$ equals − 0.036, which can be interpreted as a 3.6% lower average score on the test in 2019 compared to 2012. This is only a slight decline. The results of the different competency clusters show a significant negative trend for using symbols and mathematical formalism. Students’ ability in this competency decreased by about 12% between 2012 and 2019. Concerning the tasks focusing on modeling and using representations aside from symbols, slightly positive but non-significant trends could be observed. This indicates that the curriculum reform in Germany that shifted the focus toward acquiring these competencies only had a small positive effect. A reason might be that the tasks focusing on modeling only assessed level 1 of the modeling competency (see Table 2), i.e. they at most required a mathematization of a context given and maybe an interpretation of the result, but not the use of actual modeling assumptions. Such tasks are also called word problems (Jankvist & Niss, 2020). Furthermore, concerning modeling, the score achieved in 2019 lowered the overall trend. In 2019, the students reached a mean score in modeling of 1.08, while the score rose slightly to 1.19 in 2018. Excluding this last cohort from the sample would double the trend to about 0.008, but it would remain insignificant. Nevertheless, the small increasing trends for modeling and representing indicate a slight compensation for the declining symbolic competencies, resulting in the non-significant overall test performance trend.

Interaction Effects of Grade and the Type of Higher Education Entrance Certificate

These interaction effects are shown in Table 6. This table abbreviates the type of higher education entrance certificate with EQ.

Table 6 Results of the Trend Analysis with Interaction Effects, *p < 0.1, **p < 0.05

While the pooled trend results for the overall test performance and the different competencies of our competency clusters were insignificant, we found some significant effects for certain parts of the sample. The interaction effect of trend and grade indicates that students with better mathematics grades at secondary school were getting even better over time. In other words, the gap between high- and low-performing students got wider over the last decade. Concerning modeling, the performance increased significantly ($p<0.01$) in the subsample of students with better grades but decreased significantly ($p<0.01$) for students with a general university entrance certificate. The latter phenomenon seems surprising. One explanation might be that students with a school leaving certificate for universities of applied sciences acquired this at vocational schools with a more application-oriented curriculum. Furthermore, we see a significant positive trend ($p<0.05$) in the mathematical representation competency for students with higher grades, and vice versa, a decreasing trend for students with lower grades in mathematics at school. A reason might be that stronger students might benefit more from a stronger emphasis on tasks requiring the use of different mathematical representations, which are often rather complex.

Trends in the Individual Tasks

Unlike the overall test performance and most competencies in our competency clusters, several significant trends could be observed on the level of individual tasks. These are shown in Table 7. For the other tasks, the trends were not significant.

Table 7 Tasks with significant trends in students’ performance, *p < 0.1, **p < 0.05, ***p < 0.01

Tasks with a significant negative trend were tasks 1, 2, 4, 5, 8, 10, 12, 13, 16, 19, and 23. These mainly focused on the mathematical symbols and formalism competency (tasks 1, 2, 4, 5, 8, 10, 12, 16). They, for instance, required students to simplify terms like the double fraction $\frac{3}{8}\cdot \frac{16}{5}/\frac{4}{5}$ (task 2), to solve an equation like the quadratic equation ${\left(x-2\right)}^{2}-2=-1$ (task 12), or to use powers and logarithms like determining ${\mathrm{log}}_{3}\frac{1}{9}$ (task 5). In particular, the decreasing scores of tasks 4 and 8 (power calculation) and task 5 (logarithm calculation) were about two times higher than those of tasks 1 (basic term simplification) and 2 (fractions), which indicates that students heavily lost power and logarithm calculation skills.

However, the tasks in which the students improved significantly (tasks 18, 24, 28, and 30) were more interesting. All these focused on process competencies aside from using symbols and mathematical formalism. In task 18, the students were asked to explain how the graph of $f\left(x\right)={\left(x-a\right)}^{2}+b$ with $a,b\in {\mathbb{R}}^{+}$ changes if $a$ is increasing. Hence, it was about the influence of parameters on the graph of a given function and required switching to its graphical representation (maybe just in mind). Such tasks on the effect of parameters on the graph of a function are much more typical at school nowadays—especially in connection with the use of GeoGebra—than they used to be several years ago (see, e.g., the school textbooks Griesel et al., 2016, versus Griesel et al., 1999). The same applies to task 28, where the students were asked to differentiate a function on the graphical level. In the third task with a significant improvement (task 24), the students were asked to find the point in time at which an object with the velocity $v=-4{t}^{3}+12{t}^{2}$ has the acceleration 0. This task particularly involves an interpretation of the derivative as a rate of change of a function—an issue that is strongly emphasized in the reformed curricula (KMK [Conference of the ministers of education of Germany], 2012) and that is treated in school textbooks nowadays extensively, for example in Freudigmann et al. (2012). Finally, in task 30, the students were required to justify why the sum of two odd numbers is always even. This is not a “typical school task.” An explanation of the increasing performance in this item might be that argumentation actually plays a more important role in class nowadays than it used to be formerly. Overall, the trends in the individual tasks particularly suggest that the students have improved in such tasks that are much more common at school nowadays than they used to be 10 years ago before the curriculum reform.

Discussion

This section will summarize our results and explain how these extend findings from previous studies cited in our literature review. Afterward, we will discuss some limitations of our study, yielding some starting points for future research.

Summary of the Results and Contribution to the Field

As mentioned in the section “Theoretical Background,” former research had suggested that students’ mathematical entry skills had been decreasing—overall and in different content domains (Gill et al., 2010; Hunt & Lawson, 1996; Lawson, 2003; Treacy & Faulkner, 2015; Treacy et al., 2016). We extended this research as follows:

• 1. We collected data between 2012 and 2019 from a test whose items also addressed mathematical processes emphasized in many recent curricula, such as mathematizing, mathematical reasoning, or using different representations of mathematical objects. We analyzed this data from a competency perspective (Niss & Højgaard, 2019).

• 2. Our study widened the view on the development of students’ mathematical competencies geographically since previous studies focused only on the regions of Ireland and the UK.

• 3. We did not just look at changes in the test performance over time. Instead, we determined trends with a regression, in which we included certain socio-demographic control variables that might have influenced the cohorts’ test performance over time (see Table 4) so that the results might be less biased by changes in the structure of the different cohorts. Furthermore, we included interaction effects of the grade in mathematics and the higher education entrance certificate type into the regression to control for subsample skill differences over time.

We determined these trends for students’ overall test performance, the different mathematical competencies by Niss and Højgaard (2019), and on the level of individual tasks.

Concerning the overall test, our study shows just a slight and non-significant decline in students’ overall test performance over time after controlling the variables just mentioned (see Table 5). This contrasts with most of the studies referred to in the section “Theoretical Background” that showed a decline in students’ mathematical skills. However, we found a significant negative trend in the test performance for students with lower entry grades, which indicates a rising heterogeneity among students, as reported in other studies like Hodds et al. (2020) or Treacy et al. (2016).

Concerning the different mathematical competencies by Niss and Højgaard (2019), our study first showed a significant decline in the mathematical symbol and formalism competency. This stands in line with other studies like Treacy and Faulkner (2015) or Lawson (2003), as their tests almost solely consisted of calculation questions. Hence, our study suggests that this trend is not a specific problem in the UK and Ireland but might also occur in other West-European countries. However, our data indicate that this negative trend was slightly compensated by gains in tasks focusing on other process competencies like modeling or using different representations—even if these gains were insignificant (see Table 5). This is a first indication that the curriculum reform emphasizing these mathematical process competencies might have had a small effect.

On the level of individual tasks, our data even show some significant positive trends. Examples were a task to differentiate a function graphically or a task asking how the change of the parameter $a$ in the function $f\left(x\right)={\left(x-a\right)}^{2}+b$ ($a,b\in {\mathbb{R}}^{+}$) influences its graph—both focusing on a mathematical object’s graphical representation. Another example was a task requiring an interpretation of the derivative as a rate of change, which is essential for using the derivative in modeling situations. These tasks are practiced at school nowadays explicitly due to the given curriculum reform. This suggests that a change in the type of tasks covered at school might induce a change in the competencies the students acquire at school, and that covering tasks focusing on the abovementioned process competencies more extensively at school in the future might foster the acquisition of such.

Limitations and Outlook

Finally, we want to discuss some limitations and give an ensuing outlook for further research.

• 1. Since we could not use an experimental design, the causes for the changes in students’ competencies remain speculative, for example, whether changes in the curriculum and the tasks practiced at school caused the trends observed.

• 2. Our data only originate from one particular university and one course. Gathering data from different courses and universities might help determine the generalizable extent of our findings.

• 3. We looked at mathematical competence only from a cognitive point of view, and our data cannot show whether students’ attitudes toward mathematics might have changed. Including further data gathered with suitable questionnaires might be helpful here.

The most significant limitation is, in our opinion, the test instrument used. Although the test items addressed different mathematical process competencies, its emphasis nevertheless lies on the symbolic and formalism competency (see Table 2) because the test initially focused on skills the course teacher considered as essential for success in his course (see section “Instruments and Data Collection”). Hence, the results might differ in a real competency test with tasks designed to test the acquisition of the different mathematical process competencies—particularly concerning the ones that have not been addressed sufficiently often as a core competency in our test instrument, such as mathematical reasoning or mathematical communication.

Besides these limitations, our study gives an important first indication of how students’ mathematical process competencies at entry to university might have changed within the last decade, particularly in the light of the effort put into curriculum reforms that focused on the acquisition of such.