A critical analysis of problems with the LBOTE category on the NAPLaN test
The National Assessment Program: Literacy and Numeracy (NAPLaN) is an annual literacy and numeracy test for all Australian students, and results from the test are disaggregated into a number of categories including language background other than English (LBOTE). For this and other categories, results on each section of the test are aggregated into state, territory and national means and standard deviations enabling comparison of performance. The NAPLaN data indicate that since the test began, in 2008, at a national level there is little difference between the results of LBOTE and non-LBOTE students on all domains of the test. This is a national result, and there is greater variation at state and territory level. However, these results defy a logic which might suggest that the LBOTE category will reflect the influence of English as a second language on test performance, rather suggesting that a second language background is not associated with test performance. In this paper, I will interrogate the variation in the LBOTE category, using data provided by the Queensland state education department, focusing on year 9 students who participated in the 2010 test. Using multiple regression and focusing on variables which are specifically related to language background, I will show that within the LBOTE category there is a wide variation of performance, and the LBOTE data are in fact hiding some of our most disadvantaged students. I will suggest alternative ways in which language learners could be identified to better empower policy and pedagogical responses to student needs.
KeywordsLanguage background other than English NAPLaN English as a second language/dialect Data categories Multiple regression
Since 2008, Australia has held the National Assessment Program: Literacy and Numeracy (NAPLaN) for all students in years 3, 5, 7 and 9. For each school in Australia, the results of these annual tests in literacy and numeracy, as well as demographic and financial information, are now displayed with annual updates, on a federal government website, “MySchool”. The Australian Curriculum Assessment and Reporting Authority (ACARA) administrates NAPLaN and MySchool, and is a statutory authority, receiving instruction from education ministers across federal and state governments. In Australia, the reforms represent a move to co-operative federalism in which education has shifted from state and territory control to become more firmly included in a national policy agenda. Whilst the reforms have been promoted in the name of both enabling greater equity for disadvantaged students, as well as improving the education level and thus capacity of the workforce in Australia, the enactment of the reforms enables surveillance of schools, and in turn states and territories, as they compete for significant funding through national partnership agreements, linked to improved performance in NAPLaN testing and other educational outcomes.1 Fine grained surveillance of state and territory performance is achieved through the use of categories for the classification of students, and accompanying statistical analysis of student performance within these categories. The performance of categories: boys, girls, socio-economic status (SES), Aboriginal and Torres Strait Islander students and students with a language background other than English (LBOTE) are referenced according to minimum standards of literacy and numeracy, with average (mean) scores of these groups compared across states, and territories and against a national average. Whilst standard deviations are also reported, less attention is given to the variation of scores within categories, even though variation provides important data on those students who are both excelling and struggling within a category.
This paper will investigate variation in the LBOTE category using state education data from Queensland, Australia. The definition for the category is that it captures students who speak a language other than English at home, or whose parent/s speak a language other than English at home (ACARA 2009). The LBOTE category is problematic because the average score of this category suggests that the LBOTE group are performing as well as or better than non-LBOTE students at a national level and across states and territories, though this outcome itself is an average. If the intention of NAPLaN is to improve equity within the education system it would seem commonsense that the categories used to disaggregate data would assist in this process. Logically, a category which identifies students with language backgrounds other than English could assist in identifying language need, particularly, given Australia’s multicultural and multilingual student population (Davison and McKay 2002). However, the average scores contradict any suggestion that language may impact on test performance. The variation within the category is more informative and can be analysed to identify which groups of LBOTE students may require explicit policy and pedagogical intervention. The following investigation of the category will be done using both descriptive statistics and a regression analysis of year 9 Queensland state (government) student results for the 2010 NAPLaN test, in the domains of reading and spelling.
Whilst the LBOTE category is not confined to the migrant population,2 there is an important historical education policy and debate context related to the education performance of migrant students in Australia that makes the LBOTE results more alarming. There have been considerable changes made to the funding of English language programs over the preceding decade. Originally the responsibility of the federal government because of its management of migration policy, federal funding was allocated to intensive English programs for new arrivals. However, all federally allocated government school funding for ESL programs has now been collapsed into revised funding arrangements between state and federal governments, with funding linked to improvements in the broad areas of literacy and numeracy, teacher quality and low socio-economic school communities (Lingard et al. 2012). The danger is that the invisibility of language learners in targeted reforms, and the LBOTE category is complicit here, could result in a loss of focus on some of the most disadvantaged students in the school system. The shift to a broad mainstream literacy focus, exemplified by NAPLaN, means that audit processes are silent in relation to responding to language learner needs. In addition to these policies, there has also been significant discussion about the need to specifically target migrant children for educational support. The suggestion that migrant students enjoy academic advantage has been the subject of considerable debate (Windle 2004; Kalantzis and Cope 1988; Bullivant 1988; Birrell 1987), based on research specifically focussed on post-compulsory school and university participation. Some of this debate has focussed critically on the use of broad data categories such as LBOTE and a failure to disaggregate detail in the category so that ethnicity and SES are also considered (Windle 2004).
In many ways, this paper will extend this debate by drawing on recent NAPLaN test data, and utilizing a range of variables in order to investigate what factors are significant in understanding how language background impacts on academic performance. However, the primary focus of the analysis will be to determine the association between LBOTE categorisation and NAPLaN attainment when a range of language factors are accounted for in the model. Language factors will be explored through multiple regression (Agresti and Finlay 2009; Cohen et al. 2011), which enables the use of control variables, in this case, related to gender, socio-economic factors, geographic factors, teacher-judged academic achievement and school attendance, and then isolating the effect on the LBOTE variable of other factors related to ESL identification, language background of student, length of time in Australia, and visa category. In agreement with Watkins (2008) and Smala (2012), I wish to move away from a notion of ethnicity defining student performance. Ethnicity is a constructed social category and is applied on occasion to ‘explain’ academic performance, leading to stereotypical views of particular cultural groups (Garnaut 2010; Shepherd 2010; Mansell 2011; Ferrari 2012). Instead, the analysis seeks to explore the complex intersection of factors which contribute to the academic performance of students for whom English is a second language. This analysis will be informed by research from the field of second language acquisition. It is hoped that this empirical analysis will move the debate about migrant education performance away from the reification of the “ethnic advantage” to highlighting the range of performance within the category, and suggesting possible policy and pedagogic responses.
How heterogeneous is the LBOTE category with respect to performance on NAPLaN? If the category is disaggregated, is it possible to show that there is considerable variation amongst the groups within the category?
How is the LBOTE category associated with NAPLaN results once we take account of variables associated with language?
The body of the paper will commence with a literature review outlining key research which has informed the process and construction of the analyses models. This will be followed by the analyses, first outlining the construction of the variables used, followed by some descriptive statistics pertinent to the analysis, and finally, the regression output. A short discussion will foreground the conclusions.
The empirical research findings from the field of applied linguistics suggest that the test performance of students who are second language learners is impacted by the level of their proficiency in the test language (Cummins 1981; Thomas and Collier 1997). Level of proficiency in the academic test language will be related to educational history, which will be associated with SES, years of education and quality of education (Thomas and Collier 1997; Hakuta et al. 2000; Garcia 2000; Miller and Windle 2010). This analysis does not include a direct measure of English language proficiency: these data are not collected systemically by the Queensland education department. However, the analyses will utilise a range of ‘proxy language’ measures, which target factors influential to level of English language development.
Understanding second language proficiency has been the subject of research that has mostly built upon the early work of Cummins (1981) and his theory of Basic Interpersonal Communicative Skills (BICS) and Cognitive Academic Language Proficiency (CALP). BICS refers to basic spoken English, usually acquired for survival purposes and embedded in routine social activities. The relatively simpler and faster process of acquiring BICS is differentiated from the more demanding and slower process of acquiring academic language skills commensurate with school age English speaker norms. Cummins’ research and later research (Thomas and Collier 1997; Hakuta et al. 2000; Garcia 2000) argued that on the basis of Cummins’ theory, any concept of language proficiency for school students must be in relation to CALP, which encapsulates the academic language demands, both spoken and written, of English. Cummins (1981), Thomas and Collier (1997) and Hakuta et al. (2000) analysed the performance of English language learners using mainstream English testing tools, including standardised tests. Cummins (1981), in his early research found that students took from 5 to 7 years to achieve native like proficiency in academic English. Thomas and Collier (1997) refined these findings and determined that such a time frame was possible if students had been educated in their first language for a considerable numbers of years, at least to year 6 level, and thus had achieved a high level of CALP in first language. Students who arrived at school before the age of 8 required 7–10 years of schooling to reach native speaker norms, and students who arrived as adolescents probably had insufficient time to “catch up” to their English speaking peers. The considerable length of time required to acquire academic English can be explained by the fact that whilst language learners are acquiring English, their English speaking peers are also progressing through school, further developing their own conceptual knowledge and academic language skills. These findings do not inform the strict participation requirements in NAPLaN for students from a LBOTE. These students are allowed a 12 month exemption only, from their date of arrival to Australia, regardless of their stage of language development (ACARA 2011b). This does not align with research findings which suggest that this is insufficient time to acquire English as a second or additional language (Cummins 1981; Thomas and Collier 1997; Hakuta et al. 2000; Garcia 2000). Nor is the factor of time spent learning academic English able to be disaggregated from LBOTE data.
Research with ESL students in grades 1–6 also found a strong correlation between SES and rate of language acquisition: students who attended schools with high poverty levels progressed more slowly (Hakuta et al. 2000). Further, using parent education as an indicator of SES, it was found that students whose parents had the highest levels of education performed well above the remainder of the group (Hakuta et al. 2000). However, other research findings suggest that SES can be a difficult measure to interpret for language learners. Thomas and Collier (1997) found that despite more than half of their student sample being eligible for free school lunches, and thus classified as low SES, there were differences between the student group and the average American family living in poverty which were associated with pre-settlement characteristics. They found that some immigrant families, whilst perhaps experiencing income reduction in the U.S., were well educated, and of middle class background in their countries of origin. They found that low SES status in America was confounded with other factors related to parent education levels, family aspirations and SES status in home country (Thomas and Collier 1997 p. 38). In these cases, measure of parent education level was more useful as a measure of student success, than as a measure of SES (Thomas and Collier 1997). In the regressions, parent education level is included as a control variable.
Recent Australian research on ESL students who are of migrant and refugee background prioritises the changing characteristics of this group, particularly in relation to students of refugee background and their high levels of educational need (see Brown et al. 2006; Dooley 2009; Miller and Windle 2010; Wigglesworth 2003). The commonality of language learning need is counterbalanced by factors which relate to: educational history and access to age appropriate schooling; experiences of lives disrupted by war and associated experiences of torture and trauma; and fundamental differences in the oral and literacy practices of cultures, in which some cultures are highly literate, whilst other cultures’ primary form of communication is oral (Nicholas and Williams in Wigglesworth 2003). In the analyses, visa category enables the inclusion of this classification, as a proxy but indirect measure of prior educational experience and family circumstances.
In Australia, language learners are also not confined to students who arrive to Australia from non-English speaking countries during their school years. In fact, many language learners are born in Australia and grow up in homes in which their parents speak another language or an English which does not align with academic English (McIntosh et al. 2012). Heterogeneity in the group may be associated with cultural backgrounds, types of English dialects spoken, levels of poverty, and language(s) used by parents and the surrounding community (Garcia 2000). For language background students born in Australia, the challenge in identifying them requires that they have identified the language they speak at home as a language other than English. Country of birth (Australia) is not helpful, nor is visa category, as this group would not be associated with particular visa categories.
There is no easy match between the research problems, existing research understandings and potential data categories available to explore the problem. However, the inclusion of a range of ‘language proxy’ variables about the student has been done with the purpose of controlling as far as is possible such factors as length and quality of schooling, parent education levels, and language background characteristics.
Data, variables and analysis
The data set has been provided by the Queensland Department of Education and Training and consists of 36,517 de-identified records of all state school students in years 9 for 2010. The data set is created from enrolment and academic data collected in schools and downloaded to Education Queensland Performance, Management and Reporting Branch (PMRB). Access to the dataset was negotiated with the support of the Education Queensland Strategic Policy and Research Division (SPR) and PMRB. All statistical work in the analyses of the dataset was carried out using Stata, version 11.2.
The data variables on Queensland state school enrolment forms are constructed according to guidelines approved by the Council of Australian Governments (COAG) and implemented by the Ministerial Council for Education, Early Childhood Development and Youth Affairs (MCEECYDA),3 to ensure national comparability of data for the purposes of disaggregation according to student background characteristics. These guidelines are contained within the Data Standards Manual (ACARA 2013). Those variables related to gender, language/s of student, countries of birth, school education of parents, and Indigenous status conform to the data collection requirements contained within the Data Standards Manual (ACARA 2013).
The variables used in the analysis will be described as they appear in the regression analyses, beginning with the dependent variables, followed by student background variables, and finally, language proxy variables. Any transformations to the variables will be explained, and the structure of dummy variables described.
Reading and spelling results for Queensland state school students, Year 9, 2010
Control (background) variables
Gender, geolocation and school departmental region are used as control variables in the analysis. School geolocation is divided into four categories: metropolitan, provincial, remote and very remote. Education Queensland consists of seven geographical regions, including: Far North Queensland, North Queensland, Central Queensland, Darling Downs South West, North Coast, South East and Greater Brisbane. As categorical variables, all were transformed into dichotomous (gender) or polytomous (geolocation, school region) ‘dummy’ variables. School geolocation has metropolitan as base variable and school region has Greater Brisbane (the most metropolitan of the regions) as base variable.
Aboriginal and Torres Strait Islander identification is included in the regressions. Aboriginal and Torres Strait Islander students constitute approximately 8 % of the year 9 population. The majority of these students identify as Aboriginal (2,214) and 386 students identify as Torres Strait Islander. 328 students state that they are both Aboriginal and Torres Strait Islander. These categories will be included in the analysis as control variables, also transformed into dummy variables, measured against non-Indigenous status.
Parent 1 school education has been included as a control variable. Parent 1 has been included as it had the least missing data of the parent 1 and parent 2 variables. Dummy variables have been created based on the following categories: year 12 or equivalent, Year 11 or equivalent, Year 10 or equivalent, Year 9 or equivalent or below and not stated/unknown. The reference category in this group of dummy variables is education level of year 9 or below.
Average test results for Indigenous students in both year levels were 60–70 NAPLaN points below that of non-Indigenous students, though the spread of scores for both groups was similar. In Queensland, Indigenous students were highly disadvantaged in the test;
In Queensland, LBOTE was marginally stronger in spelling than non-LBOTE, but not as strong as non-LBOTE in all other test domains. However, the range of scores for LBOTE was greater than for non-LBOTE;
Geographic data suggests that metropolitan locations for schooling were most advantageous, and conversely, the more rural and remote the location, the more disadvantaged a student was on the test;
Finally, the higher the level of education of the parent, the better the performance of the student on the test (ACARA 2010).
By including similar student background variables, these patterns of performance will be controlled for, in the analyses.
A to E English and Maths grades for Semester 1, 2010 represent the academic performance of students based on teacher judgement, during the semester in which the NAPLaN test was held. The A–E results are not based on the performance of students in the NAPLaN test in 2010, as the results for NAPLaN attainment are released to schools during the second semester of the school year. A–E levels are determined by teachers on the basis of classroom learning and corresponding assessment. The inclusion of these continuous measures creates a tougher test of the association of language proxy variables with NAPLaN attainment because much of the variation in NAPLaN is directly associated with academic performance and teacher judgement is an alternative and valid measure of such.
Each student has a grade for English and for Maths, ranging from A through to E, with A representing the highest grade. However, in the dataset there was considerable inconsistency in the range of subjects broadly grouped as English or Maths. For example, some English subjects were referred to as ESL English, English Academic Excellence, or Learning Support English. With a similar phenomenon evident in maths, the dataset suggests that the comparability of student performance is compromised, and that achievement of a C grade, may in fact not be equivalent in any school setting, but may in fact be a subjective representation of performance in a specific subject, in a specific location. Despite these problems with the variables, all English subjects were collapsed into one group and all maths subjects into a single maths group and these have been included as two continuous variables, as a measure of general academic competencies in these two subjects.
Attendance rates controls for student attendance during Semester 1, 2010. Because attendance is measured according to number of days absent, the relationship between attendance and NAPLaN performance is skewed. In order to take account of this, the variable was recoded by taking a logarithm of the original data. This logarithm of attendance is also included in the analysis. The units of measure in this continuous variable range from −5 to +5.
Language proxy variables
The LBOTE variable in this data set has come directly from ACARA NAPLaN data. Following the annual NAPLaN test, ACARA provides all education systems with data showing how each individual student is classified according to the data categories used to disaggregate NAPLaN results. This information, which was integrated into the dataset, allows systems to determine who has been identified in the broad data categories used for the testing. Unlike gender, SES, geolocation and Indigenous status, LBOTE is the only category that is not already provided directly to ACARA by schools; LBOTE classification depends on a student being identified as LBOTE on the day of the test, on the cover of the test booklet. It should be noted that in other states and territories, LBOTE status may be provided by schools to ACARA, but this is not the case in Queensland. In this dataset 1,938 or 5.3 % of year 9 students were identified as LBOTE in 2010. LBOTE will be included in the analysis as a dichotomous dummy variable with LBOTE as the reference category.
English as a Second Language (ESL) status is a relatively new variable for the Queensland education department and aligns with the definition of an ESL learner which is given in the ESL/ESD departmental procedure which states that an ESL student “has or is in the process of acquiring English as a second (or additional) language or dialect, and learning curriculum content through this language” (Queensland Department of Education, Employment and Training 2012).
ESL and LBOTE cross tabulations (percentages in parentheses)
Not LBOTE on NAPLaN
LBOTE on NAPLaN
595 (39.6) Incorrectly classified
906 (60.4) Probably correctly classified
Not identified as ESL
34,000 (97) Probably correctly classified
1,016 (2.9) Possibly correctly classified
Language spoken by the student is captured in the dataset by the inclusion of variables identifying up to three languages spoken. For the purposes of the analysis, students who identified as speaking a language other than English as first language were regrouped into a new dichotomous variable, showing that 2,609 students (7.1 % of year 9) identified themselves as speakers of languages other than English. This dichotomous dummy variable has non-English speaker as reference category.
Tabulation of students who indicate first language not English and language of country of birth
First language is language other than English
English speaking country of birth
Non English speaking country of birth
2,609 (7.1 %)
Table 3 suggests that the language of country of birth is not a reliable indicator of English language status. 1,418 students were born in an English speaking country, yet identify their first language as other than English. Country of birth is included in the analysis as a dichotomous variable grouping English speaking countries and non-English speaking countries, with non-English speaking country of birth as reference category.
Time in Australia is a categorical variable which accounts for years of schooling in Australia only. Student date of arrival to Australia was provided in the dataset as month and year only, in order to protect student identity. Using year of arrival, I was able to generate a new categorical variable, grouping students into three broad groups: born in Australia to 2001, date of arrival between 2002 and 2008, and date of arrival between 2009 and 2010. The majority of students (90.7 %) were born in Australia; 744 (2 %) arrived to Australia during 2009 and 2010. In the analyses, the reference category for this group of dummy variables is ‘born in Australia to 2001’. Whilst total years of schooling would provide a better measure of educational experience, years of schooling prior to arrival is not collected by the department centrally.
Visa category as a proxy measure of educational background and family circumstance has been included in the analysis. There were 83 visa subclasses in the data set and this variable has been used as the basis for a new variable, grouping the visa subclasses into 7 broad groups: refugee (0.4 %), family (0.4 %), business (0.8 %), skilled (0.5 %), education (0.3 %), other (0.1 %), unknown (0.03 %) and no visa (97.5 %). The unknown category was necessary for three visa codes (10, 20, 50) which may be bridging visa codes (010, 020, 050) or may be data entry errors. This variable, which broadly classifies visa groups, will be used in the regression as a polytomous dummy variable, with refugee status as base or reference category.
Figure 2 presents the mean scores for each of the variables related to language, for the 2010 NAPLaN spelling test. At a national level, the LBOTE category is generally stronger than the non-LBOTE category in spelling and Fig. 2 suggests how this may be possible, whilst still demonstrating potential hidden need within the LBOTE ‘eligible’ group. For spelling, there is no difference between the groups who are LBOTE and who are not. However, like reading, ESL status is associated with lower levels of performance compared to that of non-ESL students. For spelling, it appears to be advantageous to speak a language other than English, and to come from a non-English speaking country. In terms of date of arrival, only those students most recently arrived are experiencing a disadvantage. Visa category patterns are similar to reading, with students of refugee background again well below other visa groupings. Similarly, all three Indigenous groups have achieved slightly better mean scores in spelling than in reading, but are clearly well below non Indigenous students.
Appendix 1 provides a tabular summary of means, standard deviations and number of cases in relation to gender and each of the above categories for both the reading and spelling tests.
I am exploring how the LBOTE category is associated with NAPLaN test results through the use of multiple regression, using a model which includes a number of background and language proxy variables. I am estimating a series of equations in which a number of language proxy variables are introduced, in sequence, to determine how the association between LBOTE classification and the independent variables in the model changes, and to identify which variables seem to improve the explanatory power of the model.
Effects of language and other variables on NAPLaN reading performance for year 9 students, state schools, Queensland 2010
Log of attendance, sem 1
English A–E result, sem 1
Maths A–E result, sem 1
Not LBOTE (ref: LBOTE)
Not ESL (ref: ESL)
English speaking (ref: non English speaking)
English C of Birth (ref: non English C of Birth)
Time in Australia (ref: birth to 2001
Visa category (ref: refugee)
Adjusted R square
Analysis 1: reading, year 9
Effects of language and other variables on NAPLaN spelling performance for year 9 students, state schools, Queensland 2010
Log of attendance, sem 1
English A–E result, sem 1
Maths A–E result, sem 1
Not LBOTE (ref: LBOTE)
Not ESL (ref: ESL)
English speaking (ref: non English speaking)
English C of Birth (ref: non English C of Birth)
Time in Australia (ref: birth to 2001
Visa category (ref: refugee)
Adjusted R square
LBOTE is introduced in model 2, and the variable is constructed so that LBOTE is the reference category. In comparison to LBOTE students therefore, non-LBOTE students are advantaged by approximately 28 points, when LBOTE is the only language variable in the model. Between model 2 and model 7 the effect of the LBOTE variable declines and in model 7, with all language variables it has more than halved. In this model, non-LBOTE students enjoy an advantage of 12.7 NAPLaN points. The greatest decline in LBOTE occurs when ESL is introduced in model 3. As each additional language variables is included, LBOTE is further reduced, though only by 1–2 points.
The ESL dummy variable is constructed with ESL as the reference category, so the numbers presented describe the difference in NAPLaN scores between students who are not ESL and those who are. In model 3, there is an advantage of 23 points for students who are not ESL, in comparison to ESL students. By model 7 this has reduced in response to other language variables in the model. In model 7, the positive effect of not being ESL is 16 points compared to ESL students. The effect of ESL in this model remains marginally greater than LBOTE.
Speaking English as a first language has a small and positive association compared with not, but an English speaking country of birth has minimal impact and loses significance in model 6.
Length of time in Australia is significant. Students who have spent more time in Australia have higher average NAPLaN reading scores than students who arrive in the country more recently. Students who have lived in Australia for less than 2 years score ten points less, on average, than students born in Australia.
The final language proxy variable introduced into the model is that of visa category, and all visa category coefficients are measured in relation to refugee visa category. Having a skilled migrant visa shows a positive association, with an average score of 32 points more than students of refugee background. Business visas, family visas and no visa category are all advantageous in comparison to refugee visa, whilst education, other and unknown visas are not significant.
In model 7, the language variables in combination have 2 % explanatory value. Given that the model has controlled for academic achievement (A–E grades) and socio-economic factors, the explanatory value of the language variables is substantial.
Analysis 2: spelling
In the spelling regression models (Table 5), patterns in the control variables replicate those of reading, however, gender is significant and positive for females, by approximately 6 NAPLaN points in all models estimated. Parent education levels, Indigenity (though to a lesser extent that reading) and school locations are statistically significant in all models and reflect the broader Queensland state trends identified earlier in the chapter: better NAPLaN performance is associated with higher levels of parental education, and with metropolitan school location. The fact that most coefficients do not change substantially over the different models suggests that the associations between these factors and spelling performance are not confounded by differences in language ability.
Attendance is not statistically significant. Both English and Maths A–E results are statistically significant: for each additional grade attainment, moving from E to A, NAPLaN scores increase by 29 NAPLaN points and 15 NAPLaN points respectively.
Unlike reading, the introduction of LBOTE in Model 2 shows no impact on any of the control variables. This remains the case as each of the language variables is included in each consecutive model. LBOTE is not statistically significant for models 2, 3 and 7, and the co-efficient is small, moving between 1.3 and 4.7 NAPLaN points.
ESL is introduced in Model 3 and is statistically significant throughout all models, though with some movement between 7 and 11 NAPLaN points, with a positive coefficient of 6.5 in Model 7 for those students who are not ESL, in comparison to those who are.
The impact of speaking English is statistically significant and negative, with a regression coefficient of approximately 11 points. This is an interesting result and means that students who are speakers of English are disadvantaged by 11 NAPLaN points in comparison to those who speak languages other than English. English country of birth is statistically insignificant.
Date of arrival is only statistically significant in model 7 for those students most recently arrived, who experience a disadvantage of some 7 points in comparison to those students who have had all their schooling in Australia.
Finally, visa category is both significant and powerful in its association. All visa categories, in comparison to refugee, experience a large and positive effect on NAPLaN performance, ranging from 53 to 71 points.
Across the 7 models, with the inclusion of all language variables, there is very little increase in the statistical power of the model. For spelling, this suggests that language has less explanatory power in relation to achievement in NAPLaN.
Discussion and conclusion
The two analyses presented above serve to interrogate and deconstruct the LBOTE category, through the counting of other variables which pertain to language background. This very large dataset provided by the Queensland education department provides opportunity to do so, because of the various ways in which students are categorised within the school system. In contrast, ACARA and the national reform architecture provides minimal publicly available data categories for appraisal, despite the importance of the measurement of NAPLaN performance in representing the work of teachers, schools and education systems. LBOTE is a misleading category in this sense: I have established that the performance of students within this category varies widely, and that there are some measures which can provide guidance about this phenomenon. These unreported variables operate beyond the simplistic LBOTE classification and impact on NAPLaN performance for individual students, and on the aggregated performance of schools.
Within this dataset I have examined a number of possible ways in which language background could be influential to NAPLaN performance. I have considered country of birth, language spoken at home and length of time living in Australia. The summary descriptive statistics presented in Figs. 1 and 2 suggest that these factors have limited and variable impact on NAPLaN performance. Interpretation of these data, when informed by second language acquisition understandings, suggest that these factors in isolation are not helpful in determining language impact: students born in non-English speaking countries may be bilingual, or may still grow up with English as first language and the converse can also be true, particularly when we consider the significant numbers of students born in Australia who reach school with English as a second language. Length of time in Australia fails to measure access to English prior to arrival in Australia, and it fails to recognise those who speak English as first language on arrival to Australia.
The variable which most strongly picks up variation within the LBOTE group is that of visa category. There are approximately 90–100 NAPLaN points between the mean scores of the students within the skilled visa category compared to those in the refugee visa category for reading and spelling. For reading, the standard deviation for refugee students is around 48 NAPLaN points (see Appendix 1); for the students in all other visa categories standard deviation is greater. What this suggests is that the mean for the refugee group is a truer reflection of the reading attainment of the students in that group, but for all visa categories, other factors may also be contributing to the range of scores, and might relate to level of language proficiency, in combination with other possible factors like test taking skill, quality of schooling experience, and length of time studying English.
The regression models provide additional knowledge for the analysis, because within the models there include a range of non-language related variables with the language variables. It is helpful to consider the regression coefficient associated with variables, in order to compare the relative impact of the variables in explaining NAPLaN performance. For the reading analysis, the coefficients associated with being Indigenous and with A to E results for English and Maths are minimally affected by the introduction of proxy language variables, and are similar in size. There is a negative effect of approximately 20 points associated with being Indigenous in comparison with non-Indigenous students, and there is a positive effect of 27 points for English and 18 points for Maths, for each grade increase. The effect of not being LBOTE is equivalent to these, in model 2, at 28 NAPLaN points. Unlike Indigenous status and A–E grades however, which do not alter across the models, the impact of LBOTE is considerably reduced, to 13 NAPLaN points, once all other language proxy variables are accounted for. Of the language proxy variables, regression coefficients are greatest within the visa categories and exceed the coefficient size of A–E English grades. Students on skilled or business visas are some 30 points more advantaged than students of refugee background. This is a particularly robust finding because visa category was the final language variable introduced into the models and is therefore picking up residual variation that remains after controlling for all other variables. These numbers suggest that there are a number of factors which should be considered beyond LBOTE status, which both challenge the usefulness of the LBOTE category, because of its reduced coefficient, and which are informative about other potential issues which impact on NAPLaN performance. Because currently the NAPLaN categories do not allow this fine grained analysis, it is difficult to determine how the performance of students and schools can be interpreted with equity and understanding. Indeed, these data suggest clearly that in the reading test, to be Indigenous or to be of refugee background, and to be an ESL learner, is to be highly disadvantaged in the test.
The spelling test analyses provides an interesting foil to the reading analyses: the LBOTE category outperforms non-LBOTE in spelling across all year levels of the test, for each year the test has been held (see, for example, ACARA 2009, 2010). However, when these results are interrogated, again we can find that there are some students who are highly disadvantaged in the category and who are well hidden.
There is powerful evidence of the role language can potentially play in relation to NAPLaN performance. It is also clear that the suggestion that LBOTE students are performing as well as non-LBOTE students is illusory, and contributes to a particular myth about the association between language and educational achievement. Some students who are speakers of language other than English are performing well on NAPLaN, while others are highly disadvantaged. It is the interaction of multiple other variables, with language, which provides a less opaque representation of this variation in performance. It is to this finer grained analysis that educators need to turn for guidance in relation to specific and targeted ESL responses to learner need, if equity of outcome is to become a possibility.
Other measures of educational outcomes include improved attendance rates, improved Aboriginal and Torres Strait Islander student outcomes, and post school destination success.
The category will potentially include students of Aboriginal and Torres Strait Islander background, as well as second generation migrants, who are born in Australia, but whose parents were born overseas.
MCEECYDA has now been replaced by the Standing Council on School Education and Early Childhood (SCSEEC).
- Agresti, A., & Finlay, B. (2009). Statistical methods for the social sciences (4th ed.). New Jersey: Pearson Prentice Hall.Google Scholar
- Australian Bureau of Statistics. (2011). Major groups, Minor groups, Countries, table, 1269.0 Standard Australian Classification of Countries. Canberra: ABS.Google Scholar
- Australian Curriculum Assessment and Reporting Authority. (2009). National Assessment Program: Literacy and Numeracy Achievement in Reading, Writing, Language Conventions and Numeracy. Retrieved 4 April 2011, from http://www.NAPLAN.edu.au/verve/_resources/NAPLAN_2009_National_Report.pdf.
- Australian Curriculum Assessment and Reporting Authority. (2010). NAPLAN Achievement in Reading, Writing, Language Conventions and Numeracy; National Report for 2010. Sydney: ACARA.Google Scholar
- Australian Curriculum Assessment and Reporting Authority. (2011a). NAP National Assessment Program: Standards. Retrieved 12 April 2012, from http://www.nap.edu.au/Test_Results/How_to_interpret/Standards/index.html.
- Australian Curriculum Assessment and Reporting Authority. (2011b). NAPLAN: Student Participation. Retrieved 20 Feb 2013, from http://www.nap.edu.au/naplan/school-support/student-participation/student-participation.html.
- Australian Curriculum Assessment and Reporting Authority. (2013). Data Standards Manual: Student Background Characteristics. Retrieved 4 Feb 2013, from http://www.acara.edu.au/reporting/data_standards_manual_student_background_characte.html.
- Birrell, R. (1987). The educational achievement of non-English speaking background students and the politics of the community languages movement. Paper presented at the Economics of Immigration, Australian National University.Google Scholar
- Brown, J., Miller, J., & Mitchell, J. (2006). Interrupted schooling and the acquisition of literacy: Experiences of Sudanese refugees in Victorian secondary schools. Australian Journal of Language and Literacy, 29(2), 150–162.Google Scholar
- Cohen, L., Manion, L., & Morrison, K. (2011). Research methods in education (7th ed.). London: Routledge.Google Scholar
- Cummins, J. (1981). Bilingualism and minority children. Ontario: Ontario Institute for Studies in Education.Google Scholar
- Dooley, K. (2009). Re-thinking pedagogy for middle school students with little, no or severely interrupted schooling. English Teaching: Practice and Critique, 8(1), 5–22.Google Scholar
- Ferrari, J., (2012, February 18–19). Helping students learn key to better results. The Weekend Australian. Google Scholar
- Garcia, G. N. (2000). Lessons from research: What is the length of time it takes limited English proficient students to acquire English and succeed in an all-English classroom?. Washington: National Clearinghouse for Bilingual Education and minority Languages Affairs.Google Scholar
- Garnaut, J. (2010, December 11–12). China’s Confucian culture clash. The Sydney Morning Herald.Google Scholar
- Hakuta, K., Butler, Y. G., & Witt, D. (2000). How long does it take english learners to attain proficiency?. Santa Barbara: University of California Linguistic Minority Research Institute.Google Scholar
- Lingard, B., Creagh, S., & Vass, G. (2012). Education policy as numbers: Data categories and two Australian cases of misrecognition. Journal of Education Policy, 27(3), 315–333. Google Scholar
- Mansell, W. (2011, February 7). Hidden tigers: why do Chinese children do so well at school? The Guardian.Google Scholar
- McIntosh, S., O’Hanlon, R., & Angelo, D. (2012). The (In)visibility of “Language” within Australian educational documentation: Differentiating Language from Literacy and exploring particular ramifications for a group of “Hidden” ESL/D learners. In C. Gitsaki & R. B. Baldauf Jr (Eds.), Future directions in applied linguistics: local and global perspectives. Cambridge Scholars: Newcastle upon Tyne.Google Scholar
- Miller, J., & Windle, J. (2010). Second language literacy: putting high needs ESL learners in the frame. English in Australia, 45(3), 31–40.Google Scholar
- Nicholas, H., & Williams, A. (2003). Oracy is more than the absence of literacy: Changing learner groups in ESL classrooms in highly literate societies. In G. Wigglesworth (Ed.), The kaleidoscope of adult second language learning: Learner, teacher and researcher perspectives. Sydney: National Centre for English Language Teaching and Research, Macquarie University.Google Scholar
- Queensland Department of Education and Training. (2012). CRP-PR-013: English as a Second Language (ESL)/English as a Second Dialect (ESD). Brisbane: Retrieved 25 July 2012, from http://education.qld.gov.au/strategic/eppr/curriculum/crppr013/index.html.
- Queensland Department of Education, Training and Employment. (2011). De-identified dataset of Year 7 and Year 9 student information and NAPLAN results, Queensland 2010. Unpublished data.Google Scholar
- Shepherd, J. (2010, September 3). Social class affects white pupils’ exam results more than those of ethnic minorities—study. The Guardian. Google Scholar
- Thomas, W. P., & Collier, V. (1997). School effectiveness for language minority students. Washington: National clearinghouse for bilingual education.Google Scholar
- Watkins, M., & Noble, G. (2008). Cultural practices and learning: Diversity, discipline and dispositions in schooling. Sydney: University of Western Sydney and NSW Department of Education and Training, University of Western Sydney.Google Scholar
- Wigglesworth, G. (Ed.). (2003). The kaleidoscope of adult second language learning: Learner, teacher and researcher perspectives. Sydney: National Centre for English Language Teaching and Research, Macquarie University.Google Scholar