Resilience refers to successful adaption to situations despite risks that put someone at a disadvantage or adversity (Ungar 2005; Windle et al. 2011). In line with this general definition, academic resilience refers to the capacity of students to perform well in school despite a disadvantaged background (OECD 2011) or more precisely the heightened likelihood of success in school despite environmental adversities brought about by early traits, conditions, and experiences (Wang et al. 1994).
Since minimizing the influence of students’ background on the outcomes of schooling is a central topic for accomplishing equity in education, a better understanding of academic resilience may help policymakers and educators to support students from a disadvantaged background in improving their academic performance. However, different conceptualizations of academic resilience may result in conflicting conclusions. It is therefore crucial to ensure the validity of a definition.
Studies on academic resilience typically employ some operationalization of socio-economic status (SES) as an indicator of students’ risk or adversity, and they use some type of educational outcome as an indicator of positive adaptation (Tudor and Spray 2017). Thresholds are usually used to combine continuous SES and outcome measures into a binary variable that indicates academic resilience or non-resilience.
In the context of international large-scale assessments (ILSAs), most studies adopted a composite SES index to operationalize students’ background. General problems such as missing data or questionable comparability of this index across countries (Watermann et al. 2016), are specifically related to the conceptualization of academic resilience: A composite SES index treats student background as one-dimensional. Thus analyses based on such an index do not reveal the potential relevance of different SES components. Furthermore, studies applied different thresholds to define a disadvantaged background (and also to what it means to perform well). Whereas some studies used the same fixed thresholds for all countries included in their study, others used relative thresholds derived from the data within each country. The evidence supporting the validity of these decisions was often quite limited.
Since the measurement of academic resilience is inherently influenced by definitional issues, this study sought to examine the validity of different conceptualizations of academic resilience and how these affect the composition of academically resilient students. For this purpose, three countries were selected representing diverse cultures and performance levels (Norway, Peru, and Hong Kong). Student performance in science was used as an indicator of educational outcomes.
Besides the common composite SES index also used in other studies, three specific background indicators representing different dimensions of SES (economic, cultural, and social) were adopted to operationalize student background. Two types of thresholds (the same fixed and relative within-country thresholds) were applied to define a disadvantaged student background or high performance. Thus, in total, sixteen conceptualizations of academic resilience were examined on their validity, with four background indicators and two types of thresholds.
To illustrate how many and which students were classified as academically resilient, we selected two individual student characteristics (gender and language spoken at home). As validity measures, we selected two school-related characteristics (sense of belonging and absence from school) that can be supposed to assess similar concepts. This study examined their concurrent validity by comparing the relations of these external constructs to the different conceptualizations of academic resilience.
The paper is organized as follows. Firstly, a conceptual framework is developed that distinguishes between different ways to define academic resilience, including their underlying norms. Secondly, an overview of the literature about academic resilience is provided, in particular in the context of ILSAs. Research gaps and the research questions examined in this paper are presented thereafter. Thirdly, a methods section follows that provides information about the data and variables used and the analyses applied, results are presented after that. Finally, the paper concludes with a summary and a discussion of implications.
Conceptual framework: criteria of academic resilience and underlying norms
Resilience and academic resilience
Research on resilience in the behavioral sciences began to emerge around 1970. Since the mid-1980s, an increasing number of researchers from different disciplines (e.g., child development, pediatrics, psychology, psychiatry, and sociology) have published findings from studies on children who were successful in life despite adverse childhood environments (Werner 2000). The theoretical development about resilience has went through four waves: (1) identifying resilient qualities, (2) uncovering the resilience process, (3) promoting resilience through prevention and intervention, and (4) focusing on the dynamics of adaptation and change (Masten 2007). The latter means that resilience may vary across contexts and over time (Tudor and Spray 2017).
Although there is no universal definition for resilience across the different disciplines examining this phenomenon, most definitions are based around two core concepts: adversity and positive adaptation (Windle 2011). Correspondingly, in the context of schooling, academic resilience is defined by some measure of adversity in terms of early traits, conditions and experiences and by some measure of increased likelihood to succeed in school (Wang et al. 1994).
Measuring adversity: composite vs. distinct measures of student background
From a theoretical point of view, it is possible to distinguish between different dimensions (e.g., education, social status, and wealth) of an individual’s background that may predefine his or her chances later in life. In major theories, the effects of social background on student outcomes are therefore conceptualized not only as a consequence of material possessions but also based on social and cultural practices (Bourdieu 1986). According to Bourdieu’s capital theory (1986), individuals process economic, cultural, and social capital such as monetary resources, cultural possessions, and social relationships. These three types of capital can be distinguished, and each of them can be used for the accumulation of other types of capital.
Academic resilience studies typically use a composite index that covers several of these background dimensions. For example, the composite SES index of the PISA studies, economic, social, and cultural status (ESCS), includes parents’ occupation, parents’ education, and home resources (OECD 2017). Although the ESCS covers two of the three Bourdieu dimensions of capital, it is treated as a one-dimensional measure. Consequently, analyses conducted with this index cannot reveal the relevance of the different SES subdimensions for being academically resilient.
Studies on academic resilience using International Association for the Evaluation of Educational Achievement (IEA) data, for example TIMSS 2015, usually use the Home Educational Resources (HER) index, which is based on parents’ education, the number of books at home, and home study support (Mullis and Martin 2013). It is therefore mostly a measure of students’ cultural capital. Parents’ occupation status as an indicator of students’ economic capital was not included in the HER index, but was a part of another SES index for Grade four students, Home Resources for Learning (HRL).
Since the measurement of academic resilience is inherently influenced by conceptual issues (Windle et al. 2011), including alternative measures of social, cultural, and economic capital in the definition may shed light on how these dimensions of social background affect the results (Watermann et al. 2016). The present study follows this idea and assesses adversity with both composite and distinct measures.
A specific challenge is that the differences between countries make it challenging to use the same background measures to study academic resilience across countries (Coronado-Hijón 2017). For example, owning a car is often used as one indicator of student background, but this may have different meanings in economically developed and developing countries. Some of the measures used in the present study address this challenge; we will examine this issue further in the discussion.
Measuring positive adaptation: selecting an Indicator of student outcome
Educational outcomes can be distinguished into cognitive and non-cognitive (Heckman et al. 2006). Unlike resilience studies in psychology, non-cognitive outcomes were rarely used to measure positive adaptation in education (Tudor and Spray 2017). Non-cognitive skills like self-efficacy or educational aspiration were merely regarded as protective factors promoting academic resilience, or as outcomes of being resilient (OECD 2018). As a result, these studies tended to use cognitive outcomes, especially test scores to measure positive adaptation.
Test scores stem either from one or several subject domains. In case of using one subject domain, most studies focused on reading, mathematics, or science. These domains were regarded as providing fundamental skills needed for further education or success in the labor market (OECD 2018). Thus, one purpose of these studies was to shed light on the competitiveness of a country.
Considering that positive adaptation may vary by domain, some studies used data from different domains. OECD (2011) found students who showed positive adaptation in science did usually so also in mathematics or/and reading. However, other studies found that positive adaptation in one domain was not necessarily associated with positive adaptation in other domains. Therefore, they defined resilience as a characteristic across domains, for example, by showing positive adaptation in reading, mathematics, and sciences (Agasisti et al. 2018).
As previously stated, studies using cognitive skills to operationalize positive adaptation usually treated traits like anxiety, motivation, or engagement as predictors or outcomes associated with resilience due to bidirectional developmental processes (Coronado-Hijón 2017). Therefore, some educational researchers recently also began to use non-cognitive outcomes to assess positive adaptation in resilience studies (OECD 2018).
Thresholds for adversity and positive adaptation: cross-country vs. within-country
Despite decisions on selecting indicators of adversity and positive adaption, another step in conceptualizing academic resilience is to decide about the thresholds, which define a “disadvantaged” background (adversity) or “high” performance (positive adaptation). These decisions vary substantially across studies. One core distinction is between “fixed” and “relative” thresholds.“Fixed” means that the same threshold is applied across countries, whereas “relative” means that based on within-country data, different thresholds are used for different countries.
Using fixed thresholds stresses an international perspective where direct cross-country comparisons are at the forefront. In this perspective, the proportion of the resilient student is regarded as an indicator for quality and equity of education systems (Erberer et al. 2015; OECD 2011). Using relative thresholds means to define academic resilience from a national perspective, provides important insights on policy levers that are associated with resilience within different education systems (OECD 2011). When relative thresholds were applied, for example, successful disadvantaged students in one country may be classified as poor performing in other contexts.
A similar distinction as the one between fixed and relative thresholds is frequently made in the research on poverty that differentiates between absolute and relative poverty (Hagenaars and De Vos 1988). Research on academic resilience is more complex because it combines information from two criteria, student background and educational outcome. Therefore, we need to distinguish between four possible approaches to define academic resilience: (1) a fixed threshold for background and a fixed threshold for outcome; (2) a fixed threshold for background but a relative threshold for outcome; (3) a relative threshold for background but a fixed threshold for outcome; and (4) a relative threshold for background and a relative threshold for outcome. Several cutoff values (e.g. 20%, 25%, or 33%) were used to define thresholds in many studies; considering the economic and performance differences among our three samples, cutoff value 33% was adopted to have more students for analysis. Details are reported below.
State of research
Overview about academic resilience studies in international large-scale assessments
Since ILSAs have facilitated cross-country analyses of student achievement and its predictors, there is an increasing number of studies using data from ILSAs to investigate how individual and institutional features are related to academic resilience (Gonzalez and Padilla 1997; Martin and Marsh 2006; Sandoval-Hernández and Bialowolski 2016). To summarize the state of research, a systematic literature search in Web of Science, ERIC, and Google Scholar was carried out in July 2019. The search was built around four groups of key words: education (e.g., academic), resilience (e.g., resilient, buoyance), measurement (e.g., scale), and ILSA (e.g., PISA). The search was limited to English-language publications and revealed about 20 studies directly related to our topic (see Table 1). They applied a broad range of different criteria, different approaches to setting thresholds, and different threshold levels.
Table 1 Operationalizations of academic resilience in ILSAs Table 1 shows the different operationalizations, which were grouped according to the four approaches to set thresholds explained above. As the overview reveals, most studies used Organization for Economic Co-operation and Development (OECD) data rather than IEA data. One reason could be the tremendous influence of PISA (Meyer et al. 2017), another possible reason could be the missing data problem on student background indicators in IEA data (Broer et al. 2019). Therefore, studies using IEA data to explore academic resilience often either adopted a self-developed SES index (García-Crespo et al. 2019) or focused on selected countries with enough SES information (Cheung 2017; Erberer et al. 2015).
We will next review the ILSA research on academic resilience with respect to the conceptualizations and operationalizations used (see Table 1). For substantive results of these studies, please see the last column in this table.
Fixed background and fixed outcome thresholds
We identified three studies that used fixed thresholds to define both disadvantage and positive adaptation across different countries. Erberer et al. (2015) examined how prevalent academic resilience was across education systems and which protective factors could be identified. Their study used TIMSS 2011 data and adopted the composite Home Educational Resources (HER) index as a family SES measure. The authors classified a student as disadvantaged by applying a fixed threshold (a score ≤ 7.3 on the HER scale). Meanwhile, the authors used the so-called TIMSS International Intermediate Benchmark of Mathematics (students that reached this benchmark can apply basic mathematical knowledge in simple situations) as a threshold (a score ≥ 475) to define positive adaptation.
Sandoval-Hernández and Bialowolski (2016) adopted Erberer et al.’s (2015) method, and applied the definition to TIMSS 2011 data from five Asian education systems. Frempong et al. (2016) also followed this procedure and applied the definition to TIMSS 2011 data from South Africa. Frempong et al. did not adopt the HER index but calculated student SES index based on 18 assets listed in the student questionnaire.
In these three studies, the fixed thresholds for achievement to define positive adaptation were set either around the international (Erberer et al. 2015; Sandoval-Hernández and Bialowolski 2016) or the national mean (Frempong et al. 2016).
Fixed background and relative outcome thresholds
Our systematic review revealed only one study that adopted a fixed threshold to define adversity and a relative threshold to define positive adaptation. Sandoval-Hernández and Cortés (2012) applied the concept of academic resilience to Progress in International Reading Literacy Study (PIRLS) 2006 data. Since PIRLS 2006 does not provide a composite SES index (Mullis et al. 2004), authors followed Caro and Cortés’s method (2012) and calculated an index based on parents’ education, parents’ occupation status, and home possessions. Considering measurement invariance, authors restricted their analysis to a cluster of countries with a comparable SES index. Disadvantaged background was defined by adopting a fixed SES threshold which was the 20th percentile of the index in the pooled data of all countries in the cluster. Positive adaptation was defined by a relative threshold which was the 80th percentile in each country.
Relative background and fixed outcome thresholds
Within this approach to define academic resilience, a methodological difference was found how to use the thresholds set. These were either used directly as in the studies described above or each disadvantaged student’s performance was compared with the performance predicted by the average relationship among students from similar SES backgrounds across countries. The difference between these two was called a student’s “residual” performance. Furthermore, within this group of studies, one of them used non-cognitive skills as an indicator of educational outcomes, and two of them used an across-domain operationalization of educational outcomes.
Direct threshold approaches
OECD (2011) adopted the composite index ESCS and defined disadvantaged students by a relative background threshold (bottom 1/3 of ESCS within each country), whereas positive adaptation was defined by a fixed threshold (top 1/3 of students’ performance across countries). OECD (2017) narrowed both thresholds down by defining academically resilient students as those who were in the bottom 1/4 of ESCS within each country and performed in the top 1/4 of students across all participating education systems. OECD (2018) adopted the same operationalization.
García-Crespo et al. (2019) explored predicting factors of academic resilience in reading literacy at Grade four, using PIRLS 2016 data from European Union member countries. The authors caculated their own Social, Economic, and Cultural Index (SECI) to measure student SES, based on home possession, number of books in the home, the highest academic qualifications of the parents, and the highest level of employment of the parents. Students in the bottom 25% of the SECI within each country, with a performance in the top 25% across the participating EU countries, were considered to be academically resilient.
Residual methods to calculate thresholds
OECD (2010) defined disadvantaged students as those in the bottom 1/4 of ESCS within each country, while disadvantaged students in the top 1/4 of residual performance across countries were classified as academically resilient.
Several studies adopted this residual method, although OECD (2011) itself adopted new methods in its later studies (OECD 2018). Cheung et al. (2014) applied the residual method to PISA 2009 data from four East Asian economies in reading literacy. Academically resilient students were defined as those in the bottom 1/4 of ESCS within each country who achieved the top 1/4 residual performance across countries. Cheung (2017) applied the same definition to PISA 2012 data and examined academic resilience in mathematics, and also focused on a cluster of East Asian education systems.
Agasisti and Longobardi (2014, 2017) put special emphasis on disadvantaged students in disadvantaged schools and applied their definition to a group of European countries. The authors firstly selected schools among the 1/3 bottom of ESCS within each country based on the aggregated school ESCS average. From these schools, they selected those students who were in the 1/3 bottom of ESCS within the country. Resilient students were defined as disadvantaged students from disadvantaged schools who have a residual performance among the top 1/3 across countries.
Studies in this group usually focused on a cluster of countries with comparable economic and cultural backgrounds, because the strength between SES and performance varied across countries.
Across-domain operationalization of educational outcomes
The studies mentioned above included only one domain (reading, science, or mathematics) as an indicator of positive adaptation. Agasisti et al. (2018) were the first to examine academic resilience across the three core domains in PISA—reading, mathematics, and science. Academically resilient students were defined as those among the bottom 1/4 of ESCS within each country, who performed at or above Proficiency Level 3 (i.e., one above the baseline level of proficiency needed to participate in society) in all three PISA domains. OECD (2018) adopted the same operationalization.
Outcome definition including non-cognitive characteristics
Most studies in the ILSA context used cognitive outcomes (e.g., school achievement) to define positive adaptation, whereas non-cognitive skills (e.g., motivation) were treated as protective factors rather than indicators of positive adaptation. OECD (2018) examined for the first time non-cognitive outcomes and defined resilience in a non-cognitive way. Disadvantaged students from the bottom 1/4 of the ESCS distribution within each country were considered to be “socially and emotionally resilient”, if they were satisfied with their life, felt socially integrated at school, and did not suffer from test anxiety (OECD 2018). When this definition was applied, lower shares of resilient students were found in the top-performing Asian educational systems than with the application of a cognitive outcome definition.
Relative background and relative outcome thresholds
Our systematic review identified four studies applying relative thresholds to defining both adversity and positive adaptations. As OECD (2011) mentioned, the purpose was to support policy makers and stakeholders with knowledge about how to foster resilience within their education systems. Disadvantaged students were defined by a relative threshold (bottom 1/3 of ESCS within each country), and the threshold for performance was also set as a relative one (top 1/3 within each country). Karklina (2012) used the same operationalization with PISA 2006 data from Latvia. Aydiner and Kalender (2015) adopted this approach as well but changed the cutoff values for thresholds—bottom 1/4 ESCS within a country for the background threshold and top 1/4 among the disadvantaged students within a country for the performance threshold. OECD (2018) followed this approach in its study about resilience from a national perspective and classified students from the bottom 1/4 of the ESCS distribution within each country and a performance among the top 1/4 of science within each country as resilient.
Does the conceptualization of academic resilience matter: a question of validity
In summary, validity may be defined as the extent to which we can back up the inferences drawn from an assessment by arguments based on evidence (Kane et al. 2005). The present study investigates the criterion validity of different conceptualizations of academic resilience, which means their relation to external criteria. Concurrent validity, where both the actual construct and the criterion measures are supposed to assess the same underlying trait and are collected at the same time, is a core dimension of criterion validity (Cohen and Swerdlik 2018).
Concurrent validity is demonstrated when a measure is positively or negatively correlated with another relevant measure as hypothesized, or when a new measure is associated with one that was already considered valid (Fink 2010). Two external criteria that should be strongly associated with academic resilience were applied, namely sense of belonging (positively) and absence from class (negatively), both of which have been identified in the literature as predictors of academic resilience (Sandoval-Hernández and Bialowolski 2016; Tommaso et al. 2018). A sense of belonging influences student outcomes via its effects on motivation and engagement, which were considered predictors of academic resilience in many studies (Aydiner and Kalender 2015; OECD 2011). Similarly, studies revealed that students who did not frequently skip class were more likely to be resilient (OECD 2018). The purpose of our study is to examine whether the strength of these relations varied by conceptualization of resilience.
Furthermore, we applied two background characteristics often used in the literature to describe the groups of students classified as academically resilient, namely gender and the language spoken at home (Cheung et al. 2014; OECD 2011) with the purpose to see how the group compositions changes depending on the conceptualization.
The aim of our study is to examine how different conceptualizations of academic resilience affect which students (gender and language) are classified as resilient, and to what extent the conceptualizations correspond with the two external criteria (sense of belonging and absence from school). Furthermore, given that the operationalization of student background may be affected by cultural differences and student performance varies substantially across countries, we examined concurrent validity for countries representing different cultures and performance levels.