Skip to main content

Heterogeneity, school-effects and the North/South achievement gap in Italian secondary education: evidence from a three-level mixed model

Abstract

With the aim of assessing the extent of the differences in the context of Italian educational system, the paper applies multilevel modeling to a new administrative dataset, containing detailed information for more than 500,000 students at grade 6 in the year 2011/2012, provided by the Italian Institute for the Evaluation of Educational System. Data are grouped by classes, schools and geographical areas. Different models for each area are fitted, in order to properly address the heteroscedasticity of the phenomenon. The results show that it is possible to estimate statistically significant “school effects”, i.e., the positive/negative association of attending a specific school and the student’s test score, after a case-mix adjustment. Therefore, the paper’s most important message is that school effects are different in terms of magnitude and types in the three geographical macro areas (Northern, Central and Southern Italy) and are dependent on specific students’ and schools’ characteristics.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Notes

  1. 1.

    Indeed, we are aware that the quality of schools is characterized by a wider set of dimensions (such as non-cognitive achievements, school climate, cognitive test scores in additional subjects, teachers’ satisfaction, etc.). Thus, while the formulation “schools with higher tests” is somehow less intuitive, it is definitely more adequate for describing the real aspect that we are monitoring in this research. However to ease notation we use the word quality in the text.

  2. 2.

    A recent paper (Agasisti and Falzetti 2013) also showed that schools in the South practice a within-school segmentation (e.g., between classes) stronger than their counterparts in the North. In this paper, we explore between-schools differences, but we are aware that similar mechanisms (that is to say, differential effects on achievement between classes of the same school) are also operating within schools.

  3. 3.

    An interesting example of study of longitudinal data at student level in Italy is presented in Bartolucci et al. (2011).

  4. 4.

    While the distribution of \(b_j\) is checked later, its characteristics of being exogenous (in the sense of Steele et al. 2007) is not empirically verifiable in this setting.

  5. 5.

    Let us remind that the number of schools is much lower in Central Italy because the administrative classification of North, Centre and South used does not separate the country in three equal parts; instead, Central Italy only includes four regions out of twenty (Tuscany, Lazio, Marche, Umbria).

References

  1. Agasisti T, Vittadini G (2012) Regional economic disparities as determinants of students’ achievement in Italy. Res Appl Econ 4(1):33–53

    Google Scholar 

  2. Agasisti T, Falzetti P (2013) Between-classes sorting within schools and test scores: an empirical analysis of the Italian junior secondary schools. INVALSI working paper no. 20/2013

  3. Bartolucci F, Pennoni F, Vittadini G (2011) Assessment of school performance through a multilevel latent Markov Rasch model. J Educ Behav Stat 36:491–522

    Article  Google Scholar 

  4. Bertoni M, Brunello G, Rocco L (2013) When the cat is near, the mice won’t play: the effect of external examiners in the Italian schools. J Public Econ 104(1):65–77

    Article  Google Scholar 

  5. Bowers AJ, Sprott R (2012) Examining the multiple trajectories associated with dropping out of high school: a growth mixture model analysis. J Educ Res 105(3):176–195

    Article  Google Scholar 

  6. Bratti M, Checchi D, Filippin D (2007) Geographical differences in Italian students’ mathematical competencies: evidence from PISA 2003. G Econ Ann Econ 66(3):299–333

    Google Scholar 

  7. Brunello G, Checchi D (2005) School quality and family background in Italy. Econ Educ Rev 24(5):563–577

    Article  Google Scholar 

  8. Cipollone P, Montanaro P, Sestito P (2010) Value-added measures in Italian high schools: problems and findings. G Econ Ann Econ 69(2):81–114

    Google Scholar 

  9. Conti E, Duranti S, Maitino ML, Rampichini C, Sciclone N (2013) The future has early roots. Learning outcomes and school’s effectiveness in Tuscany’s primary education system. Workshop youth and their future: work, education and health, October 17th–18th, 2013, University of Salerno

  10. De Simone G, Gavosto A (2013) Patterns of value-added creation in the transition from primary to lower secondary education in Italy. Paper presented at the XXVIII national conference of labour economics, Rome, September 2013

  11. Di Liberto A (2008) Education and Italian regional development. Econ Educ Rev 27(1):94–107

    Article  Google Scholar 

  12. Goldstein H (2011) Multilevel statistical models, 4th edn. Arnold, London

    MATH  Google Scholar 

  13. Goldstein H, Sammons P (1997) The influence of secondary and junior schools on sixteen year examination performance: a cross-classified multilevel analysis. Sch Eff Sch Improv 8:219–230

    Article  Google Scholar 

  14. Goldstein H, Rasbash J, Yang M, Woodhouse G, Pan H, Nuttall D, Thomas S (1993) A multilevel analysis of school examination results. Oxf Rev Educ 19:425–433

    Article  Google Scholar 

  15. Goldstein H, Browne W, Rasbash J (2002) Partitioning variation in multilevel models. Underst Stat 1(4):223–231

    Article  Google Scholar 

  16. Goldstein H, Carpenter JR, Browne WJ (2014) Fitting multilevel multivariate models with missing data in responses and covariates that may include interactions and non-linear terms. J R Stat Soc Ser A 177:553–564

    MathSciNet  Article  Google Scholar 

  17. Gorard S, Hordosy R, Siddiqui N (2012) How unstable are ’school effects’ assessed by a value-added technique? Int Educ Stud 6(1):1

    Article  Google Scholar 

  18. Grieco N, Ieva F, Paganoni AM (2012) Performance assessment using mixed effects models: a case study on coronary patient care. IMA J Manag Math 23(2):117–131

    Article  MATH  Google Scholar 

  19. Haveman R, Wolfe B (1995) The determinants of children’s attainments: a review of methods and findings. J Econ Lit 33(4):1829–1878

    Google Scholar 

  20. Heckman JJ, Kautz T (2012) Hard evidence on soft skills. Labour Econ 19(4):451–464

    Article  Google Scholar 

  21. Ieva F, Paganoni AM (2015) Detecting and visualizing outliers in provider profiling via funnel plots and mixed effect models. Health Care Manag Sci 18:166–172

    Article  Google Scholar 

  22. Leckie G, Goldstein H (2009) The limitations of using school league tables to inform school choice. J R Stat Soc Ser A 172(4):835–851

    MathSciNet  Article  Google Scholar 

  23. Little R, Rubin D (2002) Statistical analysis with missing data. Wiley, London

    Book  MATH  Google Scholar 

  24. Masci C, Ieva F, Agasisti T, Paganoni AM (2016) Does class matter more than school? Evidence from a multilevel statistical analysis on Italian junior secondary school students. Socio-Econ Plan Sci 54:47–57

    Article  Google Scholar 

  25. Mohammed MA, Deeks JJ (2008) In the context of performance monitoring, the caterpillar plot should be mothballed in favor of the funnel plot. Ann Thorac Surg 86:348

    Article  Google Scholar 

  26. OECD (2010) PISA 2009 results: what students know and can do. OECD, Paris

  27. OECD (2013) PISA 2012 results in focus: what 15-year-olds know and what they can do with what they know. OECD, Paris

  28. Paccagnella M, Sestito P (2014) School cheating and social capital. Educ Econ 22(4):367–388

    Article  Google Scholar 

  29. Perry L, McConney A (2010) Does the SES of the school matter? An examination of socioeconomic status and student achievement using PISA 2003. Teach Coll Rec 112(4):7–8

    Google Scholar 

  30. Pigott T (2001) A review of methods for missing data. Educ Res Eval 7(4):353–383

    Article  Google Scholar 

  31. Pinheiro J, Bates D, DebRoy S, Sarkar D, The R Development Core Team (2013) Nlme: linear and nonlinear mixed effects models. R package version 3.1-111

  32. Plewis I (2011) Contextual variations in ethnic group differences in educational attainments. J R Stat Soc Ser A 174(2):419–437

    MathSciNet  Article  Google Scholar 

  33. R Development Core Team (2015) R: a language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. [Online] http://www.R-project.org

  34. Rasbash J, Leckie G, Pillinger R, Jenkins J (2011) Children’s educational progress: partitioning family, school and area effects. J R Stat Soc Ser A 173(3):419–437

    MathSciNet  Google Scholar 

  35. Snijders T, Bosker R (2012) Multilevel analysis: an introduction to basic and advanced multilevel modeling, 2nd edn. Sage, London

    MATH  Google Scholar 

  36. Spiegelhalter DJ (2002) Funnel plots for institutional comparisons (letter). Qual Saf Health Care 11:390–391

    Article  Google Scholar 

  37. Spiegelhalter DJ (2005) Funnel plots for comparing institutional performance. Stat Med 24:1185–1202

    MathSciNet  Article  Google Scholar 

  38. Steele F, Vignoles A, Jenkins A (2007) The effect of school resources on pupil attainment: a multilevel simultaneous equation modelling approach. J R Stat Soc Ser A 170(3):801–824

    MathSciNet  Article  Google Scholar 

  39. Struyf A, Hubert M, Rousseeuw PJ (1996) Clustering in an object-oriented environment. J Stat Softw 1(4):1–30

    Article  Google Scholar 

  40. Willms JD, Raudenbush SW (1989) A longitudinal hierarchical linear model for estimating school effects and their stability. J Educ Meas 26(3):209–232

    Article  Google Scholar 

Download references

Acknowledgments

This work is within FARB—Public Management Research: Health and Education Systems Assessment, funded by Politecnico di Milano. The authors are grateful to Invalsi for having provided the original dataset, and P. Falzetti for the statistical assistance in building the specific database used in this paper.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Francesca Ieva.

Appendix

Appendix

As suggested by an anonymous referee, we report the estimates of model (1), fitted discarding the CMS5, for each geographical areas, in two cases: the entire database and the reduced one. We report only the estimates of fixed effects to ease comparison between the two cases (see Table 7). This enforces us in relying the listwise deletion approach, despite the discrepancies highlighted in Sect. 2.2.

Table 7 ML estimates of fixed effects (with SEs) for model (1) without CMS5, fitted both to entire dataset (ED) and to reduced one (RD) of Northern, Central and Southern area

There are no significant differences in the two cases, so we can claim that the large proportion of missing data is not a big problem.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Agasisti, T., Ieva, F. & Paganoni, A.M. Heterogeneity, school-effects and the North/South achievement gap in Italian secondary education: evidence from a three-level mixed model. Stat Methods Appl 26, 157–180 (2017). https://doi.org/10.1007/s10260-016-0363-x

Download citation

Keywords

  • Child development
  • Multilevel models
  • School effectiveness
  • Value-added model
  • Contextual effects
  • Education