The Australian Educational Researcher

, Volume 41, Issue 1, pp 1–23 | Cite as

A critical analysis of problems with the LBOTE category on the NAPLaN test

Article

Abstract

The National Assessment Program: Literacy and Numeracy (NAPLaN) is an annual literacy and numeracy test for all Australian students, and results from the test are disaggregated into a number of categories including language background other than English (LBOTE). For this and other categories, results on each section of the test are aggregated into state, territory and national means and standard deviations enabling comparison of performance. The NAPLaN data indicate that since the test began, in 2008, at a national level there is little difference between the results of LBOTE and non-LBOTE students on all domains of the test. This is a national result, and there is greater variation at state and territory level. However, these results defy a logic which might suggest that the LBOTE category will reflect the influence of English as a second language on test performance, rather suggesting that a second language background is not associated with test performance. In this paper, I will interrogate the variation in the LBOTE category, using data provided by the Queensland state education department, focusing on year 9 students who participated in the 2010 test. Using multiple regression and focusing on variables which are specifically related to language background, I will show that within the LBOTE category there is a wide variation of performance, and the LBOTE data are in fact hiding some of our most disadvantaged students. I will suggest alternative ways in which language learners could be identified to better empower policy and pedagogical responses to student needs.

Keywords

Language background other than English NAPLaN English as a second language/dialect Data categories Multiple regression 

Introduction

Since 2008, Australia has held the National Assessment Program: Literacy and Numeracy (NAPLaN) for all students in years 3, 5, 7 and 9. For each school in Australia, the results of these annual tests in literacy and numeracy, as well as demographic and financial information, are now displayed with annual updates, on a federal government website, “MySchool”. The Australian Curriculum Assessment and Reporting Authority (ACARA) administrates NAPLaN and MySchool, and is a statutory authority, receiving instruction from education ministers across federal and state governments. In Australia, the reforms represent a move to co-operative federalism in which education has shifted from state and territory control to become more firmly included in a national policy agenda. Whilst the reforms have been promoted in the name of both enabling greater equity for disadvantaged students, as well as improving the education level and thus capacity of the workforce in Australia, the enactment of the reforms enables surveillance of schools, and in turn states and territories, as they compete for significant funding through national partnership agreements, linked to improved performance in NAPLaN testing and other educational outcomes.1 Fine grained surveillance of state and territory performance is achieved through the use of categories for the classification of students, and accompanying statistical analysis of student performance within these categories. The performance of categories: boys, girls, socio-economic status (SES), Aboriginal and Torres Strait Islander students and students with a language background other than English (LBOTE) are referenced according to minimum standards of literacy and numeracy, with average (mean) scores of these groups compared across states, and territories and against a national average. Whilst standard deviations are also reported, less attention is given to the variation of scores within categories, even though variation provides important data on those students who are both excelling and struggling within a category.

This paper will investigate variation in the LBOTE category using state education data from Queensland, Australia. The definition for the category is that it captures students who speak a language other than English at home, or whose parent/s speak a language other than English at home (ACARA 2009). The LBOTE category is problematic because the average score of this category suggests that the LBOTE group are performing as well as or better than non-LBOTE students at a national level and across states and territories, though this outcome itself is an average. If the intention of NAPLaN is to improve equity within the education system it would seem commonsense that the categories used to disaggregate data would assist in this process. Logically, a category which identifies students with language backgrounds other than English could assist in identifying language need, particularly, given Australia’s multicultural and multilingual student population (Davison and McKay 2002). However, the average scores contradict any suggestion that language may impact on test performance. The variation within the category is more informative and can be analysed to identify which groups of LBOTE students may require explicit policy and pedagogical intervention. The following investigation of the category will be done using both descriptive statistics and a regression analysis of year 9 Queensland state (government) student results for the 2010 NAPLaN test, in the domains of reading and spelling.

Whilst the LBOTE category is not confined to the migrant population,2 there is an important historical education policy and debate context related to the education performance of migrant students in Australia that makes the LBOTE results more alarming. There have been considerable changes made to the funding of English language programs over the preceding decade. Originally the responsibility of the federal government because of its management of migration policy, federal funding was allocated to intensive English programs for new arrivals. However, all federally allocated government school funding for ESL programs has now been collapsed into revised funding arrangements between state and federal governments, with funding linked to improvements in the broad areas of literacy and numeracy, teacher quality and low socio-economic school communities (Lingard et al. 2012). The danger is that the invisibility of language learners in targeted reforms, and the LBOTE category is complicit here, could result in a loss of focus on some of the most disadvantaged students in the school system. The shift to a broad mainstream literacy focus, exemplified by NAPLaN, means that audit processes are silent in relation to responding to language learner needs. In addition to these policies, there has also been significant discussion about the need to specifically target migrant children for educational support. The suggestion that migrant students enjoy academic advantage has been the subject of considerable debate (Windle 2004; Kalantzis and Cope 1988; Bullivant 1988; Birrell 1987), based on research specifically focussed on post-compulsory school and university participation. Some of this debate has focussed critically on the use of broad data categories such as LBOTE and a failure to disaggregate detail in the category so that ethnicity and SES are also considered (Windle 2004).

In many ways, this paper will extend this debate by drawing on recent NAPLaN test data, and utilizing a range of variables in order to investigate what factors are significant in understanding how language background impacts on academic performance. However, the primary focus of the analysis will be to determine the association between LBOTE categorisation and NAPLaN attainment when a range of language factors are accounted for in the model. Language factors will be explored through multiple regression (Agresti and Finlay 2009; Cohen et al. 2011), which enables the use of control variables, in this case, related to gender, socio-economic factors, geographic factors, teacher-judged academic achievement and school attendance, and then isolating the effect on the LBOTE variable of other factors related to ESL identification, language background of student, length of time in Australia, and visa category. In agreement with Watkins (2008) and Smala (2012), I wish to move away from a notion of ethnicity defining student performance. Ethnicity is a constructed social category and is applied on occasion to ‘explain’ academic performance, leading to stereotypical views of particular cultural groups (Garnaut 2010; Shepherd 2010; Mansell 2011; Ferrari 2012). Instead, the analysis seeks to explore the complex intersection of factors which contribute to the academic performance of students for whom English is a second language. This analysis will be informed by research from the field of second language acquisition. It is hoped that this empirical analysis will move the debate about migrant education performance away from the reification of the “ethnic advantage” to highlighting the range of performance within the category, and suggesting possible policy and pedagogic responses.

In this analysis, I will consider the following questions:
  1. (i)

    How heterogeneous is the LBOTE category with respect to performance on NAPLaN? If the category is disaggregated, is it possible to show that there is considerable variation amongst the groups within the category?

     
  2. (ii)

    How is the LBOTE category associated with NAPLaN results once we take account of variables associated with language?

     

The body of the paper will commence with a literature review outlining key research which has informed the process and construction of the analyses models. This will be followed by the analyses, first outlining the construction of the variables used, followed by some descriptive statistics pertinent to the analysis, and finally, the regression output. A short discussion will foreground the conclusions.

Literature Review

The empirical research findings from the field of applied linguistics suggest that the test performance of students who are second language learners is impacted by the level of their proficiency in the test language (Cummins 1981; Thomas and Collier 1997). Level of proficiency in the academic test language will be related to educational history, which will be associated with SES, years of education and quality of education (Thomas and Collier 1997; Hakuta et al. 2000; Garcia 2000; Miller and Windle 2010). This analysis does not include a direct measure of English language proficiency: these data are not collected systemically by the Queensland education department. However, the analyses will utilise a range of ‘proxy language’ measures, which target factors influential to level of English language development.

Understanding second language proficiency has been the subject of research that has mostly built upon the early work of Cummins (1981) and his theory of Basic Interpersonal Communicative Skills (BICS) and Cognitive Academic Language Proficiency (CALP). BICS refers to basic spoken English, usually acquired for survival purposes and embedded in routine social activities. The relatively simpler and faster process of acquiring BICS is differentiated from the more demanding and slower process of acquiring academic language skills commensurate with school age English speaker norms. Cummins’ research and later research (Thomas and Collier 1997; Hakuta et al. 2000; Garcia 2000) argued that on the basis of Cummins’ theory, any concept of language proficiency for school students must be in relation to CALP, which encapsulates the academic language demands, both spoken and written, of English. Cummins (1981), Thomas and Collier (1997) and Hakuta et al. (2000) analysed the performance of English language learners using mainstream English testing tools, including standardised tests. Cummins (1981), in his early research found that students took from 5 to 7 years to achieve native like proficiency in academic English. Thomas and Collier (1997) refined these findings and determined that such a time frame was possible if students had been educated in their first language for a considerable numbers of years, at least to year 6 level, and thus had achieved a high level of CALP in first language. Students who arrived at school before the age of 8 required 7–10 years of schooling to reach native speaker norms, and students who arrived as adolescents probably had insufficient time to “catch up” to their English speaking peers. The considerable length of time required to acquire academic English can be explained by the fact that whilst language learners are acquiring English, their English speaking peers are also progressing through school, further developing their own conceptual knowledge and academic language skills. These findings do not inform the strict participation requirements in NAPLaN for students from a LBOTE. These students are allowed a 12  month exemption only, from their date of arrival to Australia, regardless of their stage of language development (ACARA 2011b). This does not align with research findings which suggest that this is insufficient time to acquire English as a second or additional language (Cummins 1981; Thomas and Collier 1997; Hakuta et al. 2000; Garcia 2000). Nor is the factor of time spent learning academic English able to be disaggregated from LBOTE data.

Research with ESL students in grades 1–6 also found a strong correlation between SES and rate of language acquisition: students who attended schools with high poverty levels progressed more slowly (Hakuta et al. 2000). Further, using parent education as an indicator of SES, it was found that students whose parents had the highest levels of education performed well above the remainder of the group (Hakuta et al. 2000). However, other research findings suggest that SES can be a difficult measure to interpret for language learners. Thomas and Collier (1997) found that despite more than half of their student sample being eligible for free school lunches, and thus classified as low SES, there were differences between the student group and the average American family living in poverty which were associated with pre-settlement characteristics. They found that some immigrant families, whilst perhaps experiencing income reduction in the U.S., were well educated, and of middle class background in their countries of origin. They found that low SES status in America was confounded with other factors related to parent education levels, family aspirations and SES status in home country (Thomas and Collier 1997 p. 38). In these cases, measure of parent education level was more useful as a measure of student success, than as a measure of SES (Thomas and Collier 1997). In the regressions, parent education level is included as a control variable.

Recent Australian research on ESL students who are of migrant and refugee background prioritises the changing characteristics of this group, particularly in relation to students of refugee background and their high levels of educational need (see Brown et al. 2006; Dooley 2009; Miller and Windle 2010; Wigglesworth 2003). The commonality of language learning need is counterbalanced by factors which relate to: educational history and access to age appropriate schooling; experiences of lives disrupted by war and associated experiences of torture and trauma; and fundamental differences in the oral and literacy practices of cultures, in which some cultures are highly literate, whilst other cultures’ primary form of communication is oral (Nicholas and Williams in Wigglesworth 2003). In the analyses, visa category enables the inclusion of this classification, as a proxy but indirect measure of prior educational experience and family circumstances.

In Australia, language learners are also not confined to students who arrive to Australia from non-English speaking countries during their school years. In fact, many language learners are born in Australia and grow up in homes in which their parents speak another language or an English which does not align with academic English (McIntosh et al. 2012). Heterogeneity in the group may be associated with cultural backgrounds, types of English dialects spoken, levels of poverty, and language(s) used by parents and the surrounding community (Garcia 2000). For language background students born in Australia, the challenge in identifying them requires that they have identified the language they speak at home as a language other than English. Country of birth (Australia) is not helpful, nor is visa category, as this group would not be associated with particular visa categories.

There is no easy match between the research problems, existing research understandings and potential data categories available to explore the problem. However, the inclusion of a range of ‘language proxy’ variables about the student has been done with the purpose of controlling as far as is possible such factors as length and quality of schooling, parent education levels, and language background characteristics.

Data, variables and analysis

The data set has been provided by the Queensland Department of Education and Training and consists of 36,517 de-identified records of all state school students in years 9 for 2010. The data set is created from enrolment and academic data collected in schools and downloaded to Education Queensland Performance, Management and Reporting Branch (PMRB). Access to the dataset was negotiated with the support of the Education Queensland Strategic Policy and Research Division (SPR) and PMRB. All statistical work in the analyses of the dataset was carried out using Stata, version 11.2.

The data variables on Queensland state school enrolment forms are constructed according to guidelines approved by the Council of Australian Governments (COAG) and implemented by the Ministerial Council for Education, Early Childhood Development and Youth Affairs (MCEECYDA),3 to ensure national comparability of data for the purposes of disaggregation according to student background characteristics. These guidelines are contained within the Data Standards Manual (ACARA 2013). Those variables related to gender, language/s of student, countries of birth, school education of parents, and Indigenous status conform to the data collection requirements contained within the Data Standards Manual (ACARA 2013).

The variables used in the analysis will be described as they appear in the regression analyses, beginning with the dependent variables, followed by student background variables, and finally, language proxy variables. Any transformations to the variables will be explained, and the structure of dummy variables described.

Dependent variables

The first multiple regression will use NAPLaN scores for reading as the dependent variable, the second will use the spelling scores; both are numeric variables with normal distributions. Table 1 shows the mean and SD for the total group, for each test domain being analysed in this paper. Within each year level of the test, there is a national minimum standard, across a band level, and this is a key measurement against which student performance is reported. The significance of the national minimum standard is that it defines a level below which students require considerable academic support, whilst ACARA suggest that students who are placed at the minimum standard require targeted support to reach their full learning potential (ACARA 2011a). The year 9 national minimum standard is Band 6 and ranges from 470 to 520 NAPLaN points across all domains of the test. For the dataset, average scores attained for reading and spelling are above the national minimum standard.
Table 1

Reading and spelling results for Queensland state school students, Year 9, 2010

NAPLaN test

Mean

SD

N

Reading

552.4

65.7

33,344

Spelling

564.6

75

33,579

Source Queensland Department of Education, Training and Employment (QDETE), 2011

Control (background) variables

Gender, geolocation and school departmental region are used as control variables in the analysis. School geolocation is divided into four categories: metropolitan, provincial, remote and very remote. Education Queensland consists of seven geographical regions, including: Far North Queensland, North Queensland, Central Queensland, Darling Downs South West, North Coast, South East and Greater Brisbane. As categorical variables, all were transformed into dichotomous (gender) or polytomous (geolocation, school region) ‘dummy’ variables. School geolocation has metropolitan as base variable and school region has Greater Brisbane (the most metropolitan of the regions) as base variable.

Aboriginal and Torres Strait Islander identification is included in the regressions. Aboriginal and Torres Strait Islander students constitute approximately 8 % of the year 9 population. The majority of these students identify as Aboriginal (2,214) and 386 students identify as Torres Strait Islander. 328 students state that they are both Aboriginal and Torres Strait Islander. These categories will be included in the analysis as control variables, also transformed into dummy variables, measured against non-Indigenous status.

Parent 1 school education has been included as a control variable. Parent 1 has been included as it had the least missing data of the parent 1 and parent 2 variables. Dummy variables have been created based on the following categories: year 12 or equivalent, Year 11 or equivalent, Year 10 or equivalent, Year 9 or equivalent or below and not stated/unknown. The reference category in this group of dummy variables is education level of year 9 or below.

ACARA also disaggregates NAPLaN data on variables including gender, Indigenous status, geographic location (very remote, remote, provisional or metropolitan) and socio-economic variables related to parent education levels and employment groups. ACARA (2010) report the following patterns of achievement in their disaggregated categories in Queensland in 2010:
  • Average test results for Indigenous students in both year levels were 60–70 NAPLaN points below that of non-Indigenous students, though the spread of scores for both groups was similar. In Queensland, Indigenous students were highly disadvantaged in the test;

  • In Queensland, LBOTE was marginally stronger in spelling than non-LBOTE, but not as strong as non-LBOTE in all other test domains. However, the range of scores for LBOTE was greater than for non-LBOTE;

  • Geographic data suggests that metropolitan locations for schooling were most advantageous, and conversely, the more rural and remote the location, the more disadvantaged a student was on the test;

  • Finally, the higher the level of education of the parent, the better the performance of the student on the test (ACARA 2010).

By including similar student background variables, these patterns of performance will be controlled for, in the analyses.

A to E English and Maths grades for Semester 1, 2010 represent the academic performance of students based on teacher judgement, during the semester in which the NAPLaN test was held. The A–E results are not based on the performance of students in the NAPLaN test in 2010, as the results for NAPLaN attainment are released to schools during the second semester of the school year. A–E levels are determined by teachers on the basis of classroom learning and corresponding assessment. The inclusion of these continuous measures creates a tougher test of the association of language proxy variables with NAPLaN attainment because much of the variation in NAPLaN is directly associated with academic performance and teacher judgement is an alternative and valid measure of such.

Each student has a grade for English and for Maths, ranging from A through to E, with A representing the highest grade. However, in the dataset there was considerable inconsistency in the range of subjects broadly grouped as English or Maths. For example, some English subjects were referred to as ESL English, English Academic Excellence, or Learning Support English. With a similar phenomenon evident in maths, the dataset suggests that the comparability of student performance is compromised, and that achievement of a C grade, may in fact not be equivalent in any school setting, but may in fact be a subjective representation of performance in a specific subject, in a specific location. Despite these problems with the variables, all English subjects were collapsed into one group and all maths subjects into a single maths group and these have been included as two continuous variables, as a measure of general academic competencies in these two subjects.

Attendance rates controls for student attendance during Semester 1, 2010. Because attendance is measured according to number of days absent, the relationship between attendance and NAPLaN performance is skewed. In order to take account of this, the variable was recoded by taking a logarithm of the original data. This logarithm of attendance is also included in the analysis. The units of measure in this continuous variable range from −5 to +5.

Language proxy variables

The LBOTE variable in this data set has come directly from ACARA NAPLaN data. Following the annual NAPLaN test, ACARA provides all education systems with data showing how each individual student is classified according to the data categories used to disaggregate NAPLaN results. This information, which was integrated into the dataset, allows systems to determine who has been identified in the broad data categories used for the testing. Unlike gender, SES, geolocation and Indigenous status, LBOTE is the only category that is not already provided directly to ACARA by schools; LBOTE classification depends on a student being identified as LBOTE on the day of the test, on the cover of the test booklet. It should be noted that in other states and territories, LBOTE status may be provided by schools to ACARA, but this is not the case in Queensland. In this dataset 1,938 or 5.3 % of year 9 students were identified as LBOTE in 2010. LBOTE will be included in the analysis as a dichotomous dummy variable with LBOTE as the reference category.

English as a Second Language (ESL) status is a relatively new variable for the Queensland education department and aligns with the definition of an ESL learner which is given in the ESL/ESD departmental procedure which states that an ESL student “has or is in the process of acquiring English as a second (or additional) language or dialect, and learning curriculum content through this language” (Queensland Department of Education, Employment and Training 2012).

The ESL group is a subset of the LBOTE group, but not all LBOTE students are ESL. Table 2 provides a breakdown of those students identified as ESL, cross tabulated with the LBOTE category for the NAPLaN test, inclusive of all test domains. This table shows that of the total number of students identified as ESL by schools, nearly 40 % of the year 9 ESL group have been incorrectly classified as non-LBOTE in NAPLaN data. Such data challenge the usefulness of the category, and jeopardise the authenticity of the reported outcomes for LBOTE students. Again, this categorical variable will be used as a dichotomous dummy variable in the analysis, with ESL as the reference category.
Table 2

ESL and LBOTE cross tabulations (percentages in parentheses)

 

Not LBOTE on NAPLaN

LBOTE on NAPLaN

ESL identified

595 (39.6) Incorrectly classified

906 (60.4) Probably correctly classified

Not identified as ESL

34,000 (97) Probably correctly classified

1,016 (2.9) Possibly correctly classified

Language spoken by the student is captured in the dataset by the inclusion of variables identifying up to three languages spoken. For the purposes of the analysis, students who identified as speaking a language other than English as first language were regrouped into a new dichotomous variable, showing that 2,609 students (7.1 % of year 9) identified themselves as speakers of languages other than English. This dichotomous dummy variable has non-English speaker as reference category.

Country of birth was recoded using Australian Bureau of Statistics Standard Australian Classification of Countries (2011) into two new variables. The 173 countries in this variable were grouped according to 9 major world groups (ABS 2011). When all countries of birth are grouped into major world regions, approximately 93 % of the total group were born in the Oceania and Antarctica global region of the world, with the remaining approximately 7 % being born in other regions of the world. Geographic variables assist in highlighting the cultural and linguistic diversity of the group, but are not as useful if collapsed into broader geographical groupings, particularly in determining language background. More specifically, the broad ABS geographic boundaries include countries which are both English speaking and non-English speaking. Table 3 illustrates a related conundrum: students born in English speaking countries (such as Australia and New Zealand) can identify as speaking a language other than English.
Table 3

Tabulation of students who indicate first language not English and language of country of birth

 

First language is language other than English

English speaking country of birth

1,418

Non English speaking country of birth

1,190

Total

2,609 (7.1 %)

Table 3 suggests that the language of country of birth is not a reliable indicator of English language status. 1,418 students were born in an English speaking country, yet identify their first language as other than English. Country of birth is included in the analysis as a dichotomous variable grouping English speaking countries and non-English speaking countries, with non-English speaking country of birth as reference category.

Time in Australia is a categorical variable which accounts for years of schooling in Australia only. Student date of arrival to Australia was provided in the dataset as month and year only, in order to protect student identity. Using year of arrival, I was able to generate a new categorical variable, grouping students into three broad groups: born in Australia to 2001, date of arrival between 2002 and 2008, and date of arrival between 2009 and 2010. The majority of students (90.7 %) were born in Australia; 744 (2 %) arrived to Australia during 2009 and 2010. In the analyses, the reference category for this group of dummy variables is ‘born in Australia to 2001’. Whilst total years of schooling would provide a better measure of educational experience, years of schooling prior to arrival is not collected by the department centrally.

Visa category as a proxy measure of educational background and family circumstance has been included in the analysis. There were 83 visa subclasses in the data set and this variable has been used as the basis for a new variable, grouping the visa subclasses into 7 broad groups: refugee (0.4 %), family (0.4 %), business (0.8 %), skilled (0.5 %), education (0.3 %), other (0.1 %), unknown (0.03 %) and no visa (97.5 %). The unknown category was necessary for three visa codes (10, 20, 50) which may be bridging visa codes (010, 020, 050) or may be data entry errors. This variable, which broadly classifies visa groups, will be used in the regression as a polytomous dummy variable, with refugee status as base or reference category.

Descriptive Statistics

Figure 1 presents the spread of mean scores for each of the language proxy variables to be used in the analysis. Visually, the graph gives an indication of how mean scores are distributed in each of the categories in the variables.
Fig. 1

Reading mean scores for year 9 students by language proxy variables (QDETE 2011)

In Fig. 1, there are 3 reference lines coming from the y axis. The top line marks the mean of the year 9 students in the Education Queensland dataset at 552.4. The broken lines at 520 and 470 represent the upper and lower limit of the national minimum standard band for year 9 students nationally. The upper level of this band is also marked on Fig. 2, which shows the variability of mean scores across the same language proxy variables, for spelling. Figures 1 and 2 show quite different placements of mean scores for the two different domains of the test. In Fig. 1, students who are identified as LBOTE score lower in the reading test than those who are not LBOTE. This is supported by the means for ESL. The influence of language (non English) and country of birth (non English) is less important; date of arrival has an impact on performance, with those coming to Australia most recently at greatest disadvantage. Visa category is clearly associated with performance, with students of refugee background performing, on average, at the lower end of the national minimum standard, whilst students from skilled migrant backgrounds are achieving results above the mean score of the total cohort. Finally all Indigenous groups are placed within the national minimum standard band and are clearly at a disadvantage in test performance in comparison with students who are not indigenous.
Fig. 2

Spelling mean scores for year 9 students by language proxy variables (QDETE 2011)

Figure 2 presents the mean scores for each of the variables related to language, for the 2010 NAPLaN spelling test. At a national level, the LBOTE category is generally stronger than the non-LBOTE category in spelling and Fig. 2 suggests how this may be possible, whilst still demonstrating potential hidden need within the LBOTE ‘eligible’ group. For spelling, there is no difference between the groups who are LBOTE and who are not. However, like reading, ESL status is associated with lower levels of performance compared to that of non-ESL students. For spelling, it appears to be advantageous to speak a language other than English, and to come from a non-English speaking country. In terms of date of arrival, only those students most recently arrived are experiencing a disadvantage. Visa category patterns are similar to reading, with students of refugee background again well below other visa groupings. Similarly, all three Indigenous groups have achieved slightly better mean scores in spelling than in reading, but are clearly well below non Indigenous students.

Appendix 1 provides a tabular summary of means, standard deviations and number of cases in relation to gender and each of the above categories for both the reading and spelling tests.

Regression Analyses

I am exploring how the LBOTE category is associated with NAPLaN test results through the use of multiple regression, using a model which includes a number of background and language proxy variables. I am estimating a series of equations in which a number of language proxy variables are introduced, in sequence, to determine how the association between LBOTE classification and the independent variables in the model changes, and to identify which variables seem to improve the explanatory power of the model.

The first model for each analysis includes all background (control) variables only. LBOTE is introduced in model two. ESL, as the category most likely to be a quasi indicator of English language learner status is introduced immediately after LBOTE. The next variables (language spoken, language of country of birth, time in Australia and visa category) are more indirect measures of language background, educational history and family circumstances and so the organisational principle upon which the models are based moves from most direct measures to most indirect measures of language. Table 4 (reading) and 5 (spelling) shows the progression of the models with some control variables (attendance, English and maths), and the staged introduction of each of the language variables. The full regressions models, including all control variables, are presented in Appendix 2 (reading) and Appendix 3 (spelling).
Table 4

Effects of language and other variables on NAPLaN reading performance for year 9 students, state schools, Queensland 2010

 

Model 1

Model 2

Model 3

Model 4

Model 5

Model 6

Model 7

Intercept

392.8

374.7

361.8

362.2

358.9

364.4

341.6

Log of attendance, sem 1

−2.2

−1.9

−1.9

−1.8

−1.8

−1.7

−1.7

English A–E result, sem 1

27.6

27.2

27.0

27.0

27.0

27.0

27.0

Maths A–E result, sem 1

17.7

17.9

17.9

18.0

18.0

18.0

17.9

Not LBOTE (ref: LBOTE)

 

27.9

18.2

14.8

13.7

13.3

12.7

Not ESL (ref: ESL)

  

22.9

20.8

18.9

17.4

16.1

English speaking (ref: non English speaking)

   

5.8**

4.5*

4.5*

4.8*

English C of Birth (ref: non English C of Birth)

    

6.8

3.1

2.3

Time in Australia (ref: birth to 2001

 2002–2008

     

−6.3

−6.3

 2009–2010

     

−10.3

−10.0

Visa category (ref: refugee)

 Family

      

17.1*

 Business

      

28.9

 Skilled

      

32.1

 Education

      

9.2

 Other

      

2.6

 Unknown

      

24.4

 No visa

      

25.3

Adjusted R square

0.388

0.396

0.399

0.399

0.399

0.400

0.401

p < 0.05; ** p < 0.01; p < 0.001. Because of the frequency of statistical significance at p < 0.001, and to reduce graphical complexity, the conventional use of *** has been replaced with . ref reference category

Analysis 1: reading, year 9

Model 1 has an explanatory power of 0.38, so explaining 38 % of the NAPLaN result. By model 7 the explanatory power of the model has increased to 40 %, with the inclusion of all language ‘proxy’ variables. While the control variables show little effect from the inclusion of language proxy variables, the only noteworthy change is a reduction in the regression coefficients of school geo-location, once LBOTE is introduced into the analysis in model 2. When LBOTE is introduced, the coefficients for school geo-location are all reduced by approximately 6 NAPLaN points suggesting that both variables are measuring student attributes which are associated with NAPLaN reading performance. (Table 4).
Table 5

Effects of language and other variables on NAPLaN spelling performance for year 9 students, state schools, Queensland 2010

 

Model 1

Model 2

Model 3

Model 4

Model 5

Model 6

Model 7

Intercept

401.6

400.8

396.9

396.2

395.9

396.8

332.7

Log of attendance, sem 1

0.3

0.4

0.4

0.3

0.3

0.4

0.5

English A–E result, sem 1

29.4

29.4

29.4

29.4

29.4

29.3

29.3

Maths A–E result, sem 1

14.8

14.8

14.8

14.7

14.7

14.7

14.7

Not LBOTE (ref: LBOTE)

 

1.4

−1.6

4.8*

4.7*

4.6*

3.8

Not ESL (ref: ESL)

  

7.0*

11.0

10.9

10.4

6.5*

English speaking (ref: non English speaking)

   

−11.0

−11.1

−11.0

−11.0

English C of Birth (ref: non English C of Birth)

    

0.5

0.1

−2.4

Time in Australia (ref: birth to 2001

 2002–2008

     

0.8

2.1

 2009–2010

     

−9.1

−7.1*

Visa category (ref: refugee)

       

 Family

      

61.3

 Business

      

65.5

 Skilled

      

63.5

 Education

      

52.7

 Other

      

56.7**

 Unknown

      

68.2*

 No visa

      

71.2

Adjusted R square

0.298

0.299

0.299

0.300

0.299

0.300

0.301

p < 0.05; ** p < 0.01; p < 0.001. Because of the frequency of statistical significance at p < 0.001, and to reduce graphical complexity, the conventional use of *** has been replaced with . ref reference category

LBOTE is introduced in model 2, and the variable is constructed so that LBOTE is the reference category. In comparison to LBOTE students therefore, non-LBOTE students are advantaged by approximately 28 points, when LBOTE is the only language variable in the model. Between model 2 and model 7 the effect of the LBOTE variable declines and in model 7, with all language variables it has more than halved. In this model, non-LBOTE students enjoy an advantage of 12.7 NAPLaN points. The greatest decline in LBOTE occurs when ESL is introduced in model 3. As each additional language variables is included, LBOTE is further reduced, though only by 1–2 points.

The ESL dummy variable is constructed with ESL as the reference category, so the numbers presented describe the difference in NAPLaN scores between students who are not ESL and those who are. In model 3, there is an advantage of 23 points for students who are not ESL, in comparison to ESL students. By model 7 this has reduced in response to other language variables in the model. In model 7, the positive effect of not being ESL is 16 points compared to ESL students. The effect of ESL in this model remains marginally greater than LBOTE.

Speaking English as a first language has a small and positive association compared with not, but an English speaking country of birth has minimal impact and loses significance in model 6.

Length of time in Australia is significant. Students who have spent more time in Australia have higher average NAPLaN reading scores than students who arrive in the country more recently. Students who have lived in Australia for less than 2 years score ten points less, on average, than students born in Australia.

The final language proxy variable introduced into the model is that of visa category, and all visa category coefficients are measured in relation to refugee visa category. Having a skilled migrant visa shows a positive association, with an average score of 32 points more than students of refugee background. Business visas, family visas and no visa category are all advantageous in comparison to refugee visa, whilst education, other and unknown visas are not significant.

In model 7, the language variables in combination have 2 % explanatory value. Given that the model has controlled for academic achievement (A–E grades) and socio-economic factors, the explanatory value of the language variables is substantial.

Analysis 2: spelling

In the spelling regression models (Table 5), patterns in the control variables replicate those of reading, however, gender is significant and positive for females, by approximately 6 NAPLaN points in all models estimated. Parent education levels, Indigenity (though to a lesser extent that reading) and school locations are statistically significant in all models and reflect the broader Queensland state trends identified earlier in the chapter: better NAPLaN performance is associated with higher levels of parental education, and with metropolitan school location. The fact that most coefficients do not change substantially over the different models suggests that the associations between these factors and spelling performance are not confounded by differences in language ability.

Attendance is not statistically significant. Both English and Maths A–E results are statistically significant: for each additional grade attainment, moving from E to A, NAPLaN scores increase by 29 NAPLaN points and 15 NAPLaN points respectively.

Unlike reading, the introduction of LBOTE in Model 2 shows no impact on any of the control variables. This remains the case as each of the language variables is included in each consecutive model. LBOTE is not statistically significant for models 2, 3 and 7, and the co-efficient is small, moving between 1.3 and 4.7 NAPLaN points.

ESL is introduced in Model 3 and is statistically significant throughout all models, though with some movement between 7 and 11 NAPLaN points, with a positive coefficient of 6.5 in Model 7 for those students who are not ESL, in comparison to those who are.

The impact of speaking English is statistically significant and negative, with a regression coefficient of approximately 11 points. This is an interesting result and means that students who are speakers of English are disadvantaged by 11 NAPLaN points in comparison to those who speak languages other than English. English country of birth is statistically insignificant.

Date of arrival is only statistically significant in model 7 for those students most recently arrived, who experience a disadvantage of some 7 points in comparison to those students who have had all their schooling in Australia.

Finally, visa category is both significant and powerful in its association. All visa categories, in comparison to refugee, experience a large and positive effect on NAPLaN performance, ranging from 53 to 71 points.

Across the 7 models, with the inclusion of all language variables, there is very little increase in the statistical power of the model. For spelling, this suggests that language has less explanatory power in relation to achievement in NAPLaN.

Discussion and conclusion

The two analyses presented above serve to interrogate and deconstruct the LBOTE category, through the counting of other variables which pertain to language background. This very large dataset provided by the Queensland education department provides opportunity to do so, because of the various ways in which students are categorised within the school system. In contrast, ACARA and the national reform architecture provides minimal publicly available data categories for appraisal, despite the importance of the measurement of NAPLaN performance in representing the work of teachers, schools and education systems. LBOTE is a misleading category in this sense: I have established that the performance of students within this category varies widely, and that there are some measures which can provide guidance about this phenomenon. These unreported variables operate beyond the simplistic LBOTE classification and impact on NAPLaN performance for individual students, and on the aggregated performance of schools.

Within this dataset I have examined a number of possible ways in which language background could be influential to NAPLaN performance. I have considered country of birth, language spoken at home and length of time living in Australia. The summary descriptive statistics presented in Figs. 1 and 2 suggest that these factors have limited and variable impact on NAPLaN performance. Interpretation of these data, when informed by second language acquisition understandings, suggest that these factors in isolation are not helpful in determining language impact: students born in non-English speaking countries may be bilingual, or may still grow up with English as first language and the converse can also be true, particularly when we consider the significant numbers of students born in Australia who reach school with English as a second language. Length of time in Australia fails to measure access to English prior to arrival in Australia, and it fails to recognise those who speak English as first language on arrival to Australia.

The variable which most strongly picks up variation within the LBOTE group is that of visa category. There are approximately 90–100 NAPLaN points between the mean scores of the students within the skilled visa category compared to those in the refugee visa category for reading and spelling. For reading, the standard deviation for refugee students is around 48 NAPLaN points (see Appendix 1); for the students in all other visa categories standard deviation is greater. What this suggests is that the mean for the refugee group is a truer reflection of the reading attainment of the students in that group, but for all visa categories, other factors may also be contributing to the range of scores, and might relate to level of language proficiency, in combination with other possible factors like test taking skill, quality of schooling experience, and length of time studying English.

The regression models provide additional knowledge for the analysis, because within the models there include a range of non-language related variables with the language variables. It is helpful to consider the regression coefficient associated with variables, in order to compare the relative impact of the variables in explaining NAPLaN performance. For the reading analysis, the coefficients associated with being Indigenous and with A to E results for English and Maths are minimally affected by the introduction of proxy language variables, and are similar in size. There is a negative effect of approximately 20 points associated with being Indigenous in comparison with non-Indigenous students, and there is a positive effect of 27 points for English and 18 points for Maths, for each grade increase. The effect of not being LBOTE is equivalent to these, in model 2, at 28 NAPLaN points. Unlike Indigenous status and A–E grades however, which do not alter across the models, the impact of LBOTE is considerably reduced, to 13 NAPLaN points, once all other language proxy variables are accounted for. Of the language proxy variables, regression coefficients are greatest within the visa categories and exceed the coefficient size of A–E English grades. Students on skilled or business visas are some 30 points more advantaged than students of refugee background. This is a particularly robust finding because visa category was the final language variable introduced into the models and is therefore picking up residual variation that remains after controlling for all other variables. These numbers suggest that there are a number of factors which should be considered beyond LBOTE status, which both challenge the usefulness of the LBOTE category, because of its reduced coefficient, and which are informative about other potential issues which impact on NAPLaN performance. Because currently the NAPLaN categories do not allow this fine grained analysis, it is difficult to determine how the performance of students and schools can be interpreted with equity and understanding. Indeed, these data suggest clearly that in the reading test, to be Indigenous or to be of refugee background, and to be an ESL learner, is to be highly disadvantaged in the test.

The spelling test analyses provides an interesting foil to the reading analyses: the LBOTE category outperforms non-LBOTE in spelling across all year levels of the test, for each year the test has been held (see, for example, ACARA 2009, 2010). However, when these results are interrogated, again we can find that there are some students who are highly disadvantaged in the category and who are well hidden.

There is powerful evidence of the role language can potentially play in relation to NAPLaN performance. It is also clear that the suggestion that LBOTE students are performing as well as non-LBOTE students is illusory, and contributes to a particular myth about the association between language and educational achievement. Some students who are speakers of language other than English are performing well on NAPLaN, while others are highly disadvantaged. It is the interaction of multiple other variables, with language, which provides a less opaque representation of this variation in performance. It is to this finer grained analysis that educators need to turn for guidance in relation to specific and targeted ESL responses to learner need, if equity of outcome is to become a possibility.

Footnotes

  1. 1.

    Other measures of educational outcomes include improved attendance rates, improved Aboriginal and Torres Strait Islander student outcomes, and post school destination success.

  2. 2.

    The category will potentially include students of Aboriginal and Torres Strait Islander background, as well as second generation migrants, who are born in Australia, but whose parents were born overseas.

  3. 3.

    MCEECYDA has now been replaced by the Standing Council on School Education and Early Childhood (SCSEEC).

References

  1. Agresti, A., & Finlay, B. (2009). Statistical methods for the social sciences (4th ed.). New Jersey: Pearson Prentice Hall.Google Scholar
  2. Australian Bureau of Statistics. (2011). Major groups, Minor groups, Countries, table, 1269.0 Standard Australian Classification of Countries. Canberra: ABS.Google Scholar
  3. Australian Curriculum Assessment and Reporting Authority. (2009). National Assessment Program: Literacy and Numeracy Achievement in Reading, Writing, Language Conventions and Numeracy. Retrieved 4 April 2011, from http://www.NAPLAN.edu.au/verve/_resources/NAPLAN_2009_National_Report.pdf.
  4. Australian Curriculum Assessment and Reporting Authority. (2010). NAPLAN Achievement in Reading, Writing, Language Conventions and Numeracy; National Report for 2010. Sydney: ACARA.Google Scholar
  5. Australian Curriculum Assessment and Reporting Authority. (2011a). NAP National Assessment Program: Standards. Retrieved 12 April 2012, from http://www.nap.edu.au/Test_Results/How_to_interpret/Standards/index.html.
  6. Australian Curriculum Assessment and Reporting Authority. (2011b). NAPLAN: Student Participation. Retrieved 20 Feb 2013, from http://www.nap.edu.au/naplan/school-support/student-participation/student-participation.html.
  7. Australian Curriculum Assessment and Reporting Authority. (2013). Data Standards Manual: Student Background Characteristics. Retrieved 4 Feb 2013, from http://www.acara.edu.au/reporting/data_standards_manual_student_background_characte.html.
  8. Birrell, R. (1987). The educational achievement of non-English speaking background students and the politics of the community languages movement. Paper presented at the Economics of Immigration, Australian National University.Google Scholar
  9. Brown, J., Miller, J., & Mitchell, J. (2006). Interrupted schooling and the acquisition of literacy: Experiences of Sudanese refugees in Victorian secondary schools. Australian Journal of Language and Literacy, 29(2), 150–162.Google Scholar
  10. Bullivant, B. M. (1988). The ethnic success ethic challenges conventional wisdom about immigrant disadvantages in education. Australian Journal of Education, 32(2), 223–243.CrossRefGoogle Scholar
  11. Cohen, L., Manion, L., & Morrison, K. (2011). Research methods in education (7th ed.). London: Routledge.Google Scholar
  12. Cummins, J. (1981). Bilingualism and minority children. Ontario: Ontario Institute for Studies in Education.Google Scholar
  13. Davison, C., & McKay, P. (2002). Counting and dis-counting learner group variation. Journal of Asian Pacific Communication, 12(1), 77–94.CrossRefGoogle Scholar
  14. Dooley, K. (2009). Re-thinking pedagogy for middle school students with little, no or severely interrupted schooling. English Teaching: Practice and Critique, 8(1), 5–22.Google Scholar
  15. Ferrari, J., (2012, February 18–19). Helping students learn key to better results. The Weekend Australian. Google Scholar
  16. Garcia, G. N. (2000). Lessons from research: What is the length of time it takes limited English proficient students to acquire English and succeed in an all-English classroom?. Washington: National Clearinghouse for Bilingual Education and minority Languages Affairs.Google Scholar
  17. Garnaut, J. (2010, December 11–12). China’s Confucian culture clash. The Sydney Morning Herald.Google Scholar
  18. Hakuta, K., Butler, Y. G., & Witt, D. (2000). How long does it take english learners to attain proficiency?. Santa Barbara: University of California Linguistic Minority Research Institute.Google Scholar
  19. Kalantzis, M., & Cope, B. (1988). Why we need multicultural education: A review of the ‘ethnic disadvantage ‘ debate. Journal of Intercultural Studies, 9(1), 39–57.CrossRefGoogle Scholar
  20. Lingard, B., Creagh, S., & Vass, G. (2012). Education policy as numbers: Data categories and two Australian cases of misrecognition. Journal of Education Policy, 27(3), 315–333. Google Scholar
  21. Mansell, W. (2011, February 7). Hidden tigers: why do Chinese children do so well at school? The Guardian.Google Scholar
  22. McIntosh, S., O’Hanlon, R., & Angelo, D. (2012). The (In)visibility of “Language” within Australian educational documentation: Differentiating Language from Literacy and exploring particular ramifications for a group of “Hidden” ESL/D learners. In C. Gitsaki & R. B. Baldauf Jr (Eds.), Future directions in applied linguistics: local and global perspectives. Cambridge Scholars: Newcastle upon Tyne.Google Scholar
  23. Miller, J., & Windle, J. (2010). Second language literacy: putting high needs ESL learners in the frame. English in Australia, 45(3), 31–40.Google Scholar
  24. Nicholas, H., & Williams, A. (2003). Oracy is more than the absence of literacy: Changing learner groups in ESL classrooms in highly literate societies. In G. Wigglesworth (Ed.), The kaleidoscope of adult second language learning: Learner, teacher and researcher perspectives. Sydney: National Centre for English Language Teaching and Research, Macquarie University.Google Scholar
  25. Queensland Department of Education and Training. (2012). CRP-PR-013: English as a Second Language (ESL)/English as a Second Dialect (ESD). Brisbane: Retrieved 25 July 2012, from http://education.qld.gov.au/strategic/eppr/curriculum/crppr013/index.html.
  26. Queensland Department of Education, Training and Employment. (2011). De-identified dataset of Year 7 and Year 9 student information and NAPLAN results, Queensland 2010. Unpublished data.Google Scholar
  27. Shepherd, J. (2010, September 3). Social class affects white pupils’ exam results more than those of ethnic minorities—study. The Guardian. Google Scholar
  28. Smala, S. (2012). The governmentality of reconciliation: adult education as a community relations technique in Northern Ireland. Discourse: Studies in the Cultural Politics of Education, 33(2), 283–297.CrossRefGoogle Scholar
  29. Thomas, W. P., & Collier, V. (1997). School effectiveness for language minority students. Washington: National clearinghouse for bilingual education.Google Scholar
  30. Watkins, M., & Noble, G. (2008). Cultural practices and learning: Diversity, discipline and dispositions in schooling. Sydney: University of Western Sydney and NSW Department of Education and Training, University of Western Sydney.Google Scholar
  31. Wigglesworth, G. (Ed.). (2003). The kaleidoscope of adult second language learning: Learner, teacher and researcher perspectives. Sydney: National Centre for English Language Teaching and Research, Macquarie University.Google Scholar
  32. Windle, J. (2004). The ethnic (dis)advantage debate revisited: Turkish background students in Australia. Journal of Intercultural Studies, 25(3), 271–286.CrossRefGoogle Scholar

Copyright information

© The Australian Association for Research in Education, Inc. 2013

Authors and Affiliations

  1. 1.School of EducationUniversity of QueenslandBrisbaneAustralia

Personalised recommendations