Countries may experience changes in learning outcomes between two cycles of international large-scale assessments (ILSAs) such as PISA or TIMSS. In order to be able to make correct, evidence-based decisions, it is crucial that educational policy and practice gain knowledge on what school factors are related to these changes. TIMSS data provides a unique opportunity for such investigations. TIMSS, based on the participating countries’ curricula, is a trend study with representative samples of students, and includes contextual information provided by students, their teachers, and principals (LaRoche et al., 2020).

In Norway, grade 9 students’ performance in mathematics declined by 9 points in lower secondary school from TIMSS 2015 to TIMSS 2019 (Kaarstein et al., 2020; Mullis et al., 2020). Learning gain per year in lower secondary school is smaller than for instance in primary school, and generally decreases with students’ age (Luyten et al., 2017). Olsen and Bjørnsson (2018) found that one year of learning gain from grade 8 to grade 9 in Norway is about 22 points, which is in line with the Norwegian TIMSS report (Kaarstein et al., 2020). The 9 points decline from 2015 to 2020 would hence reflect approximately a third of a year of schooling (Bergem et al., 2016; Olsen & Bjørnsson, 2018). At the same time, there are indications that several aspects of school climate, student motivation, and self-concept may have declined over the same time period (Mullis et al., 2020; Nilsen & Kaarstein, 2021; Wendelborg et al., 2020). According to previous research, school climate, student motivation, and self-concept are key predictors of student performance (e.g. Eccles & Wigfield, 2002; Thapa et al., 2013; Wang & Degol, 2015). School climate is a broad concept encompassing academic aspects such as emphasis on academic success, school belongingness, a safe environment, and absence of bullying (Wang & Degol, 2015).

However, it is an open question whether the noticed declines in school climate, student motivation, and self-concept contributed to the decline in performance in mathematics. It is important to gain insight into this, as stakeholders within educational policy and practice need this knowledge for evidence-based decisions. Students’ competence in numeracy is especially important in Norway, as this is one of the five basic competencies that are prioritized in the national curriculum along with reading-, writing-, oral- and digital competencies (UDIR, 2022).

While a randomized control trial design with representative samples would be the best approach to answer such a question, it is unethical to provide a very good school climate and motivate only one group of students. ILSAs have representative samples and trend designs (meaning it is possible to compare achievements across time at the national level), yet their cross-sectional design (i.e. the same individuals do not participate across time, the data are not longitudinal at the student level) does not allow for casual inferences. However, there are advanced methods that exploit ILSAs trend design to enable robust inferences (Gustafsson & Nilsen, 2022; Murnane & Willett, 2011; Nilsen & Gustafsson, 2014). The present study utilizes such methodology to exploit the trend design of TIMSS and investigate whether the noticed declines in school climate, student motivation, and self-concept are related to the decline in mathematics performance. The methodology provides a technique to investigate whether contextual factors may be related to changes in countries’ performance and should hence be of interest for other countries that seek explanations for changes in performance on ILSAs.

Theoretical framework. School climate, student motivation, and self-concept

School climate

School climate is a broad concept, and therefore it is common for researchers to focus on either one or a few of its aspects (Wang & Degol, 2015). Moreover, school climate is often defined differently from one study to the next (Grazia & Molinari, 2021; Nilsen & Teig, 2022). There are two major reviews of school climate (Thapa et al., 2013; Wang & Degol, 2015) and one systematic review of studies using ILSA data to investigate school climate (Nilsen & Teig, 2022). These three have reviewed a vast number of studies and concluded that a good school climate includes four aspects: 1. a safe environment, 2. sound relationships within the community, 3. a good academic environment, and 4. a good institutional environment.

A safe environment

A safe environment is reflected by a high degree of safety at school among students and employees and characterized by mutual respect, adherence to school rules and discipline, and absence of bullying (Thapa et al., 2013; Wang & Degol, 2015). Of all the participating TIMSS countries in 2015, Norway was one of the countries with the safest environment (Mullis et al., 2016a).

Safety is one of the aspects of school climate with the strongest relation to student outcomes (Mullis et al., 2016b), also in the Nordic countries (Gustafsson & Nilsen, 2016; Scherer & Nilsen, 2016). Absence of bullying is widely investigated in educational research (Nilsen & Teig, 2022) and is of great importance to student learning outcomes (Engel et al., 2009). Absence of bullying has been prioritized in Norway. Many Norwegian schools have special programs to reduce bullying such as the Olweus program (Havik & Ertesvåg, 2020; Olweus, 2010). Still, a large-scale national study has shown that bullying, especially digital bullying through social media, has increased over the last five years (Wendelborg et al., 2020).

Relationships within the community

This aspect of school climate is characterized by sound relationships and connectedness between all members of the school community (students, parents, teachers, and the leadership). High quality of interpersonal relationships is often characterized by trust, respect, and support (Hoy et al., 2006; Wang & Degol, 2015), where a positive student–teacher relationship is especially important for student achievement (Kincade et al., 2020).

Common indicators of this aspect of school climate in ILSAs are students’ sense of school belongingness and their well-being (Borgonovi, 2022; Nilsen & Teig, 2022). These are important for student outcomes (Borgonovi, 2022; Nilsen & Teig, 2022; Scherer & Nilsen, 2016; Wang & Degol, 2015), also in Norway (Nilsen, 2016).

Academic environment

Schools with a good academic climate are characterized by a priority and focus on the academic environment or emphasis on academic success (Nilsen & Gustafsson, 2014; Scherer & Nilsen, 2016; Wang & Degol, 2015). In such schools, all members of the school community, including students, parents, teachers, and the leadership, prioritize learning and have common ambitions and visions (Hoy et al., 2006; Nilsen & Gustafsson, 2014).

In 2006, a curriculum reform took place in Norway. The main goals and components of this new reform included a shift from the knowledge students were supposed to learn towards competencies they were expected to demonstrate (Sivesind, 2013). Previous research in Norway found that the implementation of the 2006 curriculum, may have led to a larger emphasis on learning and success (Sivesind, 2013) and improved learning outcome (Nilsen & Gustafsson, 2014). The pressure, enthusiasm, and attention from educational policy and society regarding the implementation of the curriculum, remained high for many years (Sivesind, 2013). The work with the implementation led to a higher degree of professional development for the teachers (Mullis et al., 2016a). However, from 2015 to 2019 a decline in teachers’ participation in professional development, and less emphasis on academic success, were seen (Mullis et al., 2020). It could be that these declines took place as the enthusiasm with the implementation of the curriculum of 2006 faded when attention and pressure from educational policy and society diminished.

Institutional environment

A good institutional environment is reflected by adequate access to resources (Thapa et al., 2013; Wang & Degol, 2015). In Norway, access to resources is not problematic (Mullis et al., 2020) and the difference between schools’ performance is small (Nilsen et al., 2018; OECD, 2019a).. Therefore, this aspect of school climate is of little interest in this study. Although it may explain variation across countries, it is not expected to explain variances within a resourceful, egalitarian country (Buchholtz et al., 2020).

Student motivation and self-concept

It is well known that student motivation and self-concept are important for performance (Bandura, 1997; Deci & Ryan, 1985; Eccles & Wigfield, 2002, 2020; Marsh et al., 2019). A relation between student motivation and self-concept and mathematics achievement has also been found in Norway (Jensen & Nortvedt, 2013; Kaarstein & Nilsen, 2016; Skaalvik et al., 2015).

Eccles and Wigfield (2002, 2020) developed a model called the expectancy-value model, which is the theoretical foundation behind the questions on students’ motivation and self-concept in TIMSS (Mullis et al., 2012). The model is rather complex and includes the interaction between a number of factors. However, there are three factors commonly included in educational research because these are strong predictors of student performance. The first is the interest-enjoyment value Eccles and Wigfield (2002) describes this as “the enjoyment the individual gets from performing the activity or the subjective interest the individual has in the subject” (p.120).

The second factor is the utility value, which is about “how well a particular task fits into an individual's present or future plans” (Eccles & Wigfield, 2020, p. 5). This factor, toghether with the interest-enjoyment value, is herin referred to as student motivation.

The third factor is what Eccles and Wigfield (2002) referred to as self-concept. According to Eccles and Wigfield (2002, 2020), and further elaborated by Marsh et al. (2019), self-concept is about faith in how well one will perform on a task based on past experiences and circumstances. In mathematics, self-concept is especially important for performance (Marsh et al., 2006).

There has been an increased attention to the importance of students’ motivation (interest-enjoyment value,utility value) and self-concept in the STEM (science, technology, engineering and mathematics) field as these aspects are important for individual success, long-term performance, recruitment to future education and careers in STEM, and hence for a sustainable society (Marsh et al., 2006; Regan & DeWitt, 2015; Young et al., 2018). It is therefore a concern that a large-scale national survey in Norway (“Elevundersøkelsen” or the student survey) identifies a decline in student motivation and self-concept from 2016 to 2019 for students in lower secondary school (Wendelborg et al., 2020). This survey points to declines in students’ well-being, students feeling more stressed about schoolwork and feeling more depressed as possible explanations for the decline in students’ motivation and self-concept (even if no causal evidence could be established). The decline in student motivation and self-concept is supported by findings in TIMSS (Mullis et al., 2016a, 2020).

The present study and research questions

The above review indicates that both school climate, student motivation, and self-concept may have declined from 2015 to 2019. At the same time, students’ achievement in mathematics has declined, and we hence ask the following research questions, using merged TIMSS 2015 and TIMSS 2019 data:

  1. 1.

    How has students’ motivation and self-concept in mathematics and the different aspects of school climate changed from 2015 to 2019?

  2. 2.

    What is the relation between the changes in the predictors (i.e. students’ motivation, self-concept, and the different aspects of school climate) and the changes in students’ mathematics achievement from 2015 to 2019? More specifically, we investigate whether changes in school climate, students’ motivation, and self-concept may mediate the changes in mathematics performance over time.

Methodology

Data and samples

TIMSS is an international large-scale assessment study that is repeated every fourth year. Norway has participated since the first cycle in 1995. TIMSS assesses the mathematics and science competence of students in fourth and eighth grade (fifth and ninth in Norway). Questionnaires to students, teachers and principals (and parents in grade 5) ensure collection of contextual information. About half of the more than 200 items in mathematics are trend items which are publicly unavailable and that allows for linking one cycle to the next (the same goes for science). Both these trend items, as well as the new items included in 2019, were included in the analyzes. Furthermore, the scaling of the achievements contributes to the linking of the cycles: the mean score was set to 500 points and the standard deviation (SD) to 100 points in 1995. After 1995, data from each cycle is scaled for comparability, always using the original scale with 500 as the scale midpoint and an SD of 100 (Martin et al., 2020). This is possible because many of the same countries participate in adjacent cycles, and, as mentioned above, about half of the test items are the same across two cycles. Having the same scale enables comparisons across cycles.

The sampling design is referred to as a two-stage cluster sample design, where schools are randomly picked with probability proportional to their size in the first stage, and in the second stage, two classes are picked within each school (LaRoche et al., 2020). If the school is small and only has one class, then one class is selected.

The present study includes students in grade 9 who participated in TIMSS 2015 (N = 4697) and TIMSS 2019 (N = 4575) and their mathematics teachers (N = 239 in 2015 and 277 in 2019). The mean age of the students was 14,7 in both cycles with equal percentages of girls and boys.

Measures

To cover the wide range of topics specified in the frameworks (Mullis & Martin, 2017), TIMSS has a rotating matrix design of the mathematics (and science) items. This means that each student receives only a subset of all the items in the assessment pool. However, the results from the students are sewed together to provide results at the national level in terms of five plausible values (PV). These plausible values have to be taken into account in the analyses in the correct manner, for instance through imputation in the software Mplus (see e.g. Rutkowski et al., 2013).

All variables included in the present study are described in Table 1. The variables are sorted by their function (i.e. whether they are predictors, outcomes, or control variables), and according to what they measure (e.g. aspect of school climate or aspect of student motivation). These variables are measured as latent variables including only items used in both cycles. The items covering the underlying concepts should provide valid measures of the concepts according to our theoretical framework. The variables are retrieved from student and teacher questionnaires in 2015 and 2019, available on the webpage of TIMSS (https://timssandpirls.bc.edu/).

Table 1 Overview of all latent variables included
Full size table

SES was included as latent control variable measured by the number of books at home and mother’s and father’s educational level.

Methods of analyses

The student and teacher data were merged using the software IDB analyzer (https://www.iea.nl/data-tools/tools) to ensure that the links between teachers and their students is correct. Next, the data from 2015 and 2019 were merged to one SPSS file. This data file also contains a dummy variable called Time coded to 0 for the 2015 cycle and 1 for the 2019 cycle.

The methodological approach is a mediation structural equation model (SEM) with trend data (Murnane & Willett, 2011; Nilsen & Gustafsson, 2014). Mplus version 8 (Muthén & Muthén, 1998–2017) was used to estimate a mediation SEM-model. The hierarchical design of the data, where students are nested within schools, was taken into account by using the Mplus option “TYPE = COMPLEX”, where the clustering variable is IDSCHOOL (the unique school identification). Robust maximum likelihood was used to account for missing data. The school weights and plausible values for mathematics were included according to recommended procedures (LaRoche et al., 2020; Rutkowski et al., 2013).

According to the international report, Norwegian 9th graders’ decline in mathematics achievement was about 9 points. Therefore, we expect the unstandardized regression coefficient of the effect of Time on achievement to be negative and somewhat close to 9 points (see Fig. 1 where c denotes the regression coefficient). The hypothesis is that there are other explanations for the decline in achievement than that of Time passing. Our question is whether school climate and student motivation and self-concept may mediate the effect of Time on mathematics achievement (see Fig. 2). In other words, we examine whether changes in school climate, motivation, and self-concept are related to changes in achievement.

Fig. 1
figure 1

The effect of time of achievement. The null-model

Full size image
Fig. 2
figure 2

Hypothesized mediation model in which aspects of school climate and student motivation and self-concept mediate the effect of time on achievement

Full size image

The first step of the modelling was to do a confirmatory factor analysis to examine the reliability and validity of the latent variables through inspections of factor loadings, significance of indicators, and model fit (Hox & Roberts, 2011).

The second step was to estimate a correlation matrix between all the latent variables, to check whether some were highly correlated.

Having established the reliability of the latent variables, and inspected correlations and other descriptive statistics, created a foundation for the modelling. The third step of the modelling was to estimate the effect of Time on achievement (Fig. 1, the null-model). In the fourth step, the mediation model is built. All predictors as well as the dummy variable “Time” and mathematics achievement were included in one and the same model. Including all predictors in one model means that each predictor is controlled for by all other predictors and hence allows for evaluation of which predictor is the strongest. We therefore included all predictors in one model to evaluate which was the strongest mediator (i.e. the one with the strongest indirect effect). To be able to compare the indirect effects, one needs to compare the standardized estimates.

Figure 2 shows how we examine whether the aspects of school climate, the aspects of motivation, and self-concept have changed over time (denoted by a in the figure), and whether these predictors have an effect on achievement (denoted by b). The direct effect of time on achievement (c`) is expected to decrease if the predictor mediates the time changes in achievement. The indirect effect of the predictor (a times b, not shown in Fig. 2), reflects how many points of the decrease in achievement is explained, or mediated, by the predictor. The indirect effect is the focus of research question 2. Conceptually, this reflects whether changes in predictors are related to changes in achievement. The cross-sectional data cannot tell us whether the predictors caused the decline in achievement. However, the terminology of mediation models, such as “indirect effect” is used for clarity. We would like to point out though, that we investigate relations, not effects, as these are cross-sectional data. We would further like to point out that there is a difference between: (1) merging the 2015 and 2019 data and merely estimate the regression coefficient between averaged predictor (e.g. utility value) and averaged achievement, and (2) our model that investigates whether the predictor mediates the time changes in achievement. Our model enables more robust and valid inferences than the former, even if no causal inferences may be drawn. The model does not deviate from that of a classical mediation approach, although it may be uncommon to use the Time variable as a predictor. Moreover, trend-data, rather than longitudinal data, is utilized. The trend analysis is based on country-level achievement, school climate, motivation, and self-concept data from one country (i.e., one unit of observation) and two time points. This means that a representative sample of grade 9 students in 2015 and a representative sample of grade 9 students in 2019 were included in the analyses. Hence, these are not the same students followed over time. However, TIMSS is by definition a trend-study because both achievements and contextual variables are comparable across time due to both the sampling of representative samples and the equated scales (Martin et al., 2020). This means that a significant decline in for instance achievement or school climate is not an artefact of different samples in two different cycles.

To control for SES, achievement was regressed on SES, and SES was modelled to correlate with all predictors.

Results

In the present section, we first present descriptive statistics and results from the null model. We then answer research question 1 and 2.

Table 2 shows the correlation matrix between all latent variables including all aspects of school climate, aspects of motivation, and self-concept as well as SES. Weights were applied also in this analysis. As expected, the interest-enjoyment value, utility value, and self-concept showed high correlations, and the highest was that between self-concept and the interest-enjoyment value. The only variables for which there were no significant correlation, were between the utility value and a safe environment. It should be pointed out, however, that many of the correlations were small even if they were significant. The student-reported variables well-being and bullying were correlated with the two aspects of motivation and self-concept, and they also correlated with each other. In addition, the teacher reported variables reflecting a safe environment and school emphasis on academic success were highly correlated. These correlations were expected as they were in line with previous theory (see e.g. Wang & Degol, 2016; Eccles & Wigfield, 2002).

Table 2 Correlation matrix between all latent variables
Full size table

The results for the null-model (Fig. 1) showed that the unstandardized regression coefficient of the relation between Time and mathematics achievement was -8.62 (standardized: -0.06) with a significance level of p < 0.05. This result is in line with the international TIMSS report, in which an average decline of 9 points is reported for the Norwegian students from 2015 to 2019 (Mullis et al., 2020). According to previous Norwegian results, this decline reflects about a third of a school year (Kaarstein et al., 2020; Olsen & Björnsson, 2018).

Research question 1

For research question 1, we found a significant decline from 2015 to 2019 in all variables, except for school emphasis on academic success and student interest-enjoyment value. In Table 3, the decline of the variables are shown in the second colum, with the heading «regression coefficient a». A safe environment declined significantly by -0.10 (standardized regression coefficient: -0.31). This is a moderate to large decline. The other variables exhibited small, but significant declines. Seeing how these are all latent variables, it is hard to directly interpret this in a more conceptual or practical manner. Results from descriptive statistics provided in the international TIMSS reports () are easier to interpret. To exemplify, the results from the 2015 report showed that 72% (SE = 4.0) of the Norwegian teachers felt very safe (international average was 46% with an SE = 0.5). Norway then had the second most safe environment of all participating countries. In 2019, only 53% (SE = 4.0) of the Norwegian teachers felt very safe (international average was 48% with an SE = 0.5), with 17 countries out of 39 participating countries ranking higher in terms of saftey. Safety had declined by 19%.The principals answered the same questions about safety, and the decline in their report are in line with that of the teachers.This illustrates that the decline was substantial.

Table 3 Results from the mediation models where aspects of school climate, student motivation, and self-concept are mediators of the effect of Time on achievement
Full size table

With regards to the relations between the predictors and mathematics achievement (regression coefficient b), some were significant and some were not. The variables with the largest regression regression coefficients with the highest proportion of the variance explained, were students’ self-concept (b = 48.02, standardized coefficient = 0.54) and a safe environment (b = 33.62, standardized coefficient = 0.07), which together explained about 10% of the variance. Using the example of students’ self-concept, the interpretation of the unstandardized regression coeffcient is as follows: an increase by 1 unit on the self concept scale is related to an increase of about 48 score points in mathematics achievement.

School emphasis on academic success and bullying were also significant, although the regression coefficient for bullying was small (b = 5.28, standardized coefficient = 0.03). Note that the variable bullying is reverse coded such that the decline shown in regression coefficient a, implies that bullying has increased and the regression coefficient b reflects the positive relation between absence of bullying and achievement.

School belonging and motivation were not significant.

The model presented in Table 3 had good model fit (CFI and TLI > 0.95, and RMSEA < 0.05). The results are also presented in Fig. 3 which shows the paths but not the indirect effects.

Fig. 3
figure 3

The results of the mediation model. All variables were included in the model. The indirect effects are not shown in the model

Full size image

For the latent variables, the standardized factor loadings ranged between 0.62 and 0.93 and were all significant. The model fit and factor loadings confirmed the reliability and validity of the results (Hox & Roberts, 2011).

Research question 2

For research question 2, we examined whether the different aspects of school climate, the aspects of student motivation, and self-concept could mediate the time-changes in achievement (i.e. the effect of Time on achievement). If the indirect effect (see Table 3, last column) is significant and negative, it means that the variable in question mediates the changes in achievement over time. The unstandardized indirect effect indicates how many points out of the total 8.6 points of declined achievement is explained, or mediated in the model.

Students’ self concept declined significantly from 2015 to 2019, had a significant association with achievement, and mediated the relation between Time and achievement. The indirect effect for self-concept is -5.62 (unstandardized, p < 0.05). This indicates that out of the 8.6 points of decline in mathematics achievement, 5.6 of these were mediated by student self-concept. In other words, the results indicate that students’ declining self concept is related to the declining achievement in mathematics from 2015 to 2019.

Also a safe environment declined significantly from 2015 to 2019, had a significant association with achievement, and mediated the relation between Time and achievement. The indirect effect for a safe environment is -3.24 (unstandardized, p < 0.05). This indicates that out of the 8.6 points of decline in mathematics achievement, 3.2 of these were mediated by a safe environment.

There were no significant indirect effects from the other variables. However, there is a clear pattern that all variables had negative indirect effects.

Discussion

To summarize, the key findings indicate that a safe environment and student self-concept declined from 2015 to 2019 and mediated the changes in achievement over time.

School climate

The four aspects of school climate included in the analyses were school emphasis on academic success, school belonging and well-being, bullying, and safe environment. Apart from school emphasis on academic success, the other three aspects of school climate have been found to correlate in previous studies (OECD, 2019b) which is in line with our correlation matrix (see Table 2). All three of these aspects declined from 2015 to 2019. School belonging and well-being is a latent variable that includes students’ sense of safety at school, whether the teachers are fair, whether the students feel proud of their school, like being at their school, and feel a belonging to their school. The decline in this variable is in line with findings from a large-scale national student survey that found that students’ well-being declined from 2016 to 2019 (Wendelborg et al., 2020). The decline could be related to the increased bullying (reported by students) and decrease of a safe environment (reported by teachers), especially considering the high correlation between bullying and well-being (see Table 2). Students’ sense of school belonging and well-being naturally decline in a less safe environment with increased bullying. These findings are also in line with findings from PISA (Jensen et al., 2019) and the aforementioned large-scale national student survey (Wendelborg et al., 2020). It is further in line with the national TIMSS 2019 report, although the analyses used science achievement as the outcome (Nilsen & Kaarstein, 2021).

Of the four aspects of school climate, only a safe environment mediated the changes in achievement over time. However, there is a clear pattern for the school climate variables, where all the variables decrease and exhibit negative indirect effects, albeit not significant. The reason these other variables do not quite reach significance could be the relation to other variables. Including several predictors in a model is always risky, because if the predictors are highly correlated (i.e. multi-collinearity), the significance of the relation between the predictor and outcome may be underestimated (Grewal et al., 2004). Such correlations could perhaps explain why modelling the school climate aspects one at a time, resulted in significant indirect effects (except for school emphasis), even after controlling for SES (see Additional file 1).

Whether a decline in a safe environment partly caused the declining achievements, remains an open question We found that a decline in a safe environment is related to the declining achievements. There could be omitted or confounding variables, and hence further research on this is required.

Student motivation and self-concept

Students’ utility value and self-concept in mathematics have declined from 2015 to 2019, while there was no significant change in their interest-enjoyment value. Declining motivation in general is supported by the national large-scale student survey (Elevundersøkelsen) from 2016 to 2019 (Wendelborg et al., 2020).

We did not investigate why students’ utility value and self-concept declined, while there were no significant changes in student interest-enjoyment value. It is positive that students still enjoy and like mathematics, but it is worrisome that they seem to value mathematics less and feel less confident in this subject. One explanation may be that the student composition has changed over time. Students participating in TIMSS 2019 report feeling hungrier at school, eating breakfast less often, and getting less sleep. At the same time, teachers report struggling more with students due to students increased lack of prerequisite knowledge or skills, a higher level of disruption of lessons, more uninterested students, and increased numbers of students with physical or mental disabilities (Kaarstein & Nilsen, 2021; Vik et al., 2022). Moreover, the number of students with minority background has increased extensively over the years (Lehre & Nilsen, 2021; SSB, 2021). Future studies are needed to investigate why utility value of mathematics and student self-concept have decreased and whether this is related to student characteristics and home background.

When investigating interest-enjoyment value, utility value, and self-concept, only the latter was significantly related to mathematics achievement in Norway. This is in agreement with national findings from PISA (Kjærnslie & Olsen, 2013), showing the importance of self-concept for mathematics achievement. The strong relation between self-concept and achievement, together with the high correlation between self-concept and interest-enjoyment value and utility value, may explain why only self-concept has a significant relation to achievement. Usually the values of interest-enjoyment and of utility are related to achievement (e.g. Eccles & Wigfield, 2002) too.

The findings pertaining to the second research question indicate that student self-concept not only declined, but also explained a substantial part of the declining mathematics performance from 2015 to 2019. Whenever there is a decline in a contextual variable like motivation and in achievement, it is not given that these changes are related to one another. It is hence interesting and important to survey whether these declines are in fact related, as this may provide indications on how to increase achievement. Increasing students’ self-concept may result in increased achievement. While the relationship between student motivation and achievement is known to be reciprocal (Seaton et al., 2014), the fact that student motivation decreased and affected the decreased achievement, may indicate that motivation caused decreased achievement. However, it could also be that the declining achievements caused a decline in student motivation, and hence more evidence is needed to confirm this, preferably with longitudinal data.

A mediation would require that both the relation between time and predictor, and between predictor and achievement, are significant. Only self-concept was related to achievement, and so this could explain why only self-concept mediated the time changes in achievement. Previous research has found that self-concept is more important to student achievement than utility value and interest-enjoyment value (e.g. Kjærnslie & Olsen, 2013), and this could be the reason why only self-concept (and not ulility value or interest-enjoyment value) was related to the decline. More research is needed to further investigate why there only were declines in self-concept and utility value and not in interest-enjoyment value, and why only self-concept was related to a decline in achievement.

Our findings are in line with findings from the national TIMSS report using science performance as the outcome, however, the utility value for science did not change from 2015 to 2019 (Nilsen & Kaarstein, 2021). Self-concept on the other hand, declined in both mathematics and science, and could explain substantial parts of the declines in both subject domains. This supports the finding that students’ declining self-concept could have contributed to the decline in achievement. However, it is impossible to prove a causal relation with cross-sectional data that does not have the design of a randomized controlled trial, and so further research is required to confirm these findings.

Limitations

ILSAs have cross-sectional designs, and hence no causal inferences may be drawn. However, utilizing the trend design with equal contextual scales and equal scaling of performance (mean = 500, SD = 100) across cycles, allows for more robust and valid inferences than those using cross-sectional data from one time point (Gustafsson & Nilsen, 2022; Nilsen & Gustafsson, 2014). At the same time, including more time-points in a trend analysis or using data from a true longitudinal study, would have increased the reliability and validity of the findings.

There could be other explanations for why Norway’s achievement declined from 2015 to 2019 that are not included in our analyses, such as students eating breakfast less often, and getting less sleep as discussed earlier. Furthermore, there could be confounding variables not included in the analyses that threatens causal inferences. Confounding variables is a general limitation by mediation analyses (MacKinnon et al., 2007). For instance, it could be that a decline in school safety leads to more depressed students, which again may lead to a decline in achievement. Here the confounding variable is depressed students. This was just an example, and further research would be needed to establish such causal links.

Another threat to causal inference besides that of confounding or omitted variables, are reciprocal relationships. It could for instance be that a decrease in student motivation and self-concept leads to a decline in achievement. Our data cannot establish sufficient evidence for the direction of this relationship. RCT studies are needed to confirm evidence and establish causality.

The data obtained for measuring school climate, student motivation, and self-concept are self-reported and may hence be biased. Such bias is most often problematic when comparing means across countries, since different response patterns and understandings of the concepts may result in incomparable measures (Van de Vijver & Tanzer, 1997). However, comparisons across two cycles, within the same culture or country, should reduce this bias.

Contributions, implications, and concluding remarks

A number of studies have found school climate, student motivation, and self-concept to be related to achievement in ILSAs, internationally (e.g. Eccles & Wigfield, 2020; Eklöf, 2021) and in Norway (e.g. Kaarstein & Nilsen, 2016). However, few studies have investigated how school climate, student motivation, and self-concept relate to changes in achievement. The study herein may contribute to research, both in terms of supporting findings from previous studies (e.g. Wang & Degol, 2015) and in terms of methodology; other researchers may find the method useful for investigating school factors related to change in performance. These types of models produce robust, reliable, and valid results (Gustafsson & Nilsen, 2022; Nilsen & Gustafson, 2014), something that was confirmed through model fit and the alignment between our findings and IRT analyses performed by TIMSS concerning decline in performance (Mullis et al., 2020).

Our study has shown that declines in aspects of school climate and student self-concept were related to the declines in achievements from 2015 to 2019. While evidence of a declining achievement, school climate, student motivation and self-concept in Norway is found in the literature, no study has examined whether the decline in achievement is related to a decline in school climate, student motivation and self-concept. As such, the present study adds to the literature but also opens for further research. Future research is needed to investigate whether this is a causal relationship, or whether there may be omitted variables and reciprocal relations. Moreover, other variables than the ones we identified may also have contributed to the decline.

Our findings support previous research that policy and practice should take measures to ensure a good and safe school climate, and to improve students’ self-concept, because these are pre-requisites for learning (Gustafsson & Nilsen, 2016; Scherer & Nilsen, 2016; Wang & Degol, 2015).