Two decades of havoc: A synthesis of criticism against PISA

Abstract

The Programme for International Students Assessment (PISA) has become one of the most influential forces in global education. The growing influence has been accompanied by growing criticism. For nearly two decades since the first round of PISA was conducted in 2000, the global assessment program has been roundly scrutinized and criticized by education researchers all over the world. But the mounting criticism seems to have had little impact on PISA’s influence as evidenced by its growing power in global education policy and practice. The lack of impact of criticism does not mean the criticism is not valid or PISA has improved. It simply means that the criticism has been largely ignored. The lack of impact is no reason to give up exposing PISA as a flawed business that has great power to misguide education. The expanding influence of the PISA enterprise makes it more even more important to be critical of this juggernaut today. It is also important to consider more effective and more straightforward approaches to present the criticism. The purpose of this article is to present a summary of criticisms that reveal the most fundamental flaws of PISA in non-technical language in one place. Specifically, the article focuses on criticisms of PISA’s three fundamental deficiencies: its underlying view of education, its implementation, and its interpretation and impact on education globally.

This is a preview of subscription content, access via your institution.

References

  1. Arzarello, F., Garuti, R., & Ricci, R. (2015). The impact of PISA studies on the Italian national assessment system. In K. Stacey & R. Turner (Eds.), Assessing mathematical literacy (pp. 249–260). Berlin: Springer.

    Google Scholar 

  2. Baird, J.-A., Johnson, S., Hopfenbeck, T. N., Isaacs, T., Sprague, T., Stobart, G., et al. (2016). On the supranational spell of PISA in policy. Educational Research,58(2), 121–138.

    Google Scholar 

  3. Barber, M., & Mourshed, M. (2007). How the world’s best-performing school systems come out on top. Retrieved August 3, 2010 from New York https://www.mckinsey.com/industries/social-sector/our-insights/how-the-worlds-best-performing-school-systems-come-out-on-top.

  4. Baroutsis, A., & Lingard, B. (2017). Counting and comparing school performance: An analysis of media coverage of PISA in Australia, 2000–2014. Journal of Education Policy,32(4), 432–449.

    Google Scholar 

  5. Berliner, D. C. (2011). The context for interpreting PISA results in the USA: Negativism, chauvinism, misunderstanding, and the potential to distort the educational systems of nations. In M. A. Pereyra, H.-G. Kotthoff, & R. Cowen (Eds.), Pisa under examination (pp. 77–96). New York: Springer.

    Google Scholar 

  6. Bieber, T., & Martens, K. (2011). The OECD PISA study as a soft power in education? Lessons from Switzerland and the US. European Journal of Education,46(1), 101–116. https://doi.org/10.1111/j.1465-3435.2010.01462.x.

    Article  Google Scholar 

  7. Breakspear, S. (2012). The policy impact of Pisa: An exploration of the normative effects of international benchmarking in school system performance. Retrieved July 3, 2016 from Paris http://search.oecd.org/officialdocuments/displaydocumentpdf/?cote=EDU/WKP(2012)8&docLanguage=En.

  8. Bybee, R., & McCrae, B. (2011). Scientific literacy and student attitudes: Perspectives from PISA 2006 science. International Journal of Science Education,33(1), 7–26.

    Google Scholar 

  9. Campbell, M. (2013). West vs Asia education rankings are misleading: Western schoolchildren are routinely outperformed by their Asian peers, but worrying about it is pointless. New Scientist (2898). Retrieved January 10, 2015 from https://www.newscientist.com/article/mg21728985-800-west-vs-asia-education-rankings-are-misleading/.

  10. Carney, S., Rappleye, J., & Silova, I. (2012). Between faith and science: World culture theory and comparative education. Comparative Education Review,56(3), 366–393.

    Google Scholar 

  11. Domínguez, M., Vieira, M.-J., & Vidal, J. (2012). The impact of the programme for international student assessment on academic journals. Assessment in Education: Principles, Policy & Practice,19(4), 393–409.

    Google Scholar 

  12. Eivers, E. (2010). PISA: Issues in implementation and interpretation. The Irish Journal of Education/Iris Eireannach an Oideachais,38, 94–118.

    Google Scholar 

  13. Feniger, Y., & Lefstein, A. (2014). How not to reason with PISA data: An ironic investigation. Journal of Education Policy,29(6), 845–855. https://doi.org/10.1080/02680939.2014.892156.

    Article  Google Scholar 

  14. Figazzolo, L. (2009). Impact of PISA 2006 on the education policy debate. Retrieved September 9, 2015 from http://download.ei-ie.org/docs/IRISDocuments/ResearchWebsiteDocuments/2009-00036-01-E.pdf.

  15. Friedman, T. L. (2007). The world is flat: A brief history of the twenty-first century. New York: Farrar, Straus and Giroux.

    Google Scholar 

  16. Gewertz, C. (2018). PISA working toward exam to gauge creative thinking—Built by ACT. Education Week,38(6), 4.

    Google Scholar 

  17. Goldstein, H., Bonnet, G., & Rocher, T. (2007). Multilevel structural equation models for the analysis of comparative data on educational performance. Journal of Educational and Behavioral Statistics,32(3), 252–286. https://doi.org/10.3102/1076998606298042

    Article  Google Scholar 

  18. Gould, S. J. (1996). The mismeasure of man. New York: Northon.

    Google Scholar 

  19. Green, A., & Mostafa, T. (2013). The dynamics of education systems: Convergent and divergent trends, 1990–2010. In J. Janmaat, M. Duru-Bellat, P. Méhaut, & A. Green (Eds.), The dynamics and social outcomes of education systems (pp. 15–45). Berlin: Springer.

    Google Scholar 

  20. Grek, S. (2009). Governing by numbers: The PISA ‘effect’ in Europe. Journal of Education Policy,24(1), 23–37.

    Google Scholar 

  21. Grisay, A., De Jong, J., Gebhardt, E., Berezner, A., & Halleux-Monseur, B. (2007). Translation equivalence across PISA countries. Journal of Applied Measurement,8(3), 249.

    Google Scholar 

  22. Grisay, A., & Gonzalez, E. (2009). Equivalence of item difficulties across national versions of the PIRLs and PISA reading assessments. IERI Monograph Series: Issues and Methodologies in Large-Scale Assessments,2, 63–83.

    Google Scholar 

  23. Grönqvist, E., & Vlachos, J. (2016). One size fits all? The effects of teachers’ cognitive and social abilities on student achievement. Labour Economics,42, 138–150.

    Google Scholar 

  24. Gruber, K. H. (2006). The German ‘PISA-Shock’: Some aspects of the extraordinary impact of the OECD’s PISA study on the German education system. Paper presented at the Cross-national Attraction in Education: Accounts from England and Germany. Oxford: Symposium Books.

  25. Hanushek, E. A. (2013). Economic growth in developing countries: The role of human capital. Economics of Education Review,37, 204–212.

    Google Scholar 

  26. Hanushek, E. A., & Woessmann, L. (2008). The role of cognitive skills in economic development. Journal of Economic Literature,46, 607–668.

    Google Scholar 

  27. Hanushek, E. A., & Woessmann, L. (2010). The high cost of low educational performance: The long-run economic impact of improving PISA Outcomes. Retrieved March 20, 2011 from Paris http://books.google.com/books?id=k7AGPo0NvfYC&pg=PA33&lpg=PA33&dq=hanushek+pisa+gdp&source=bl&ots=2gCfzF-f1_&sig=wwe0XLL5EblVWK9e7RJfb5MyhIU&hl=en&sa=X&ei=MLPCUqaOD8-JogS6v4C4Bw&ved=0CGcQ6AEwBjgK-v=onepage&q=hanushek%20pisa%20gdp&f=false.

  28. Hanushek, E. A., & Woessmann, L. (2012). Do better schools lead to more growth? Cognitive skills, economic outcomes, and causation. Journal of Economic Growth,17(4), 267–321.

    Google Scholar 

  29. Hargreaves, A., & Sahlberg, P. (2015). The tower of PISA is badly leaning. An argument for why it should be saved. Retrieved April 5, 2017 from https://www.washingtonpost.com/news/answer-sheet/wp/2015/03/24/the-tower-of-pisa-is-badly-leaning-an-argument-for-why-it-should-be-saved/?noredirect=on&utm_term=.69275346b5d3.

  30. Harris, A., & Jones, M. S. (2015). Leading futures: Global perspectives on educational leadership. New Delhi: SAGE Publications India.

    Google Scholar 

  31. Hatzinikita, V., Dimopoulos, K., & Christidou, V. (2008). PISA test items and school textbooks related to science: A textual comparison. Science Education,92(4), 664–687.

    Google Scholar 

  32. Hopfenbeck, T. N., Lenkeit, J., El Masri, Y., Cantrell, K., Ryan, J., & Baird, J.-A. (2018). Lessons learned from PISA: A systematic review of peer-reviewed articles on the programme for international student assessment. Scandinavian Journal of Educational Research,62(3), 333–353. https://doi.org/10.1080/00313831.2016.1258726.

    Article  Google Scholar 

  33. Hopmann, S. T. (2008). No child, no school, no state left behind: Schooling in the age of accountability. Journal of Curriculum Studies,40(4), 417–456.

    Google Scholar 

  34. Hopmann, S. T., Brinek, Gertrude, & Retzl, M. (Eds.). (2007). PISA zufolge PISA—PISA according to PISA. Berlin: Lit Verlag.

    Google Scholar 

  35. Jones, R. S. (2011). Education reform in Japan. Paris: OECD. https://doi.org/10.1787/5kg58z7g95np-en.

    Book  Google Scholar 

  36. Kamens, D. H. (2015). A maturing global testing regime meets the world economy: Test scores and economic growth, 1960–2012. Comparative Education Review,59(3), 420–446.

    Google Scholar 

  37. Kjærnsli, M., & Lie, S. (2011). Students’ preference for science careers: International comparisons based on PISA 2006. International Journal of Science Education,33(1), 121–144.

    Google Scholar 

  38. Klees, S. J. (2016). Human capital and rates of return: Brilliant ideas or ideological dead ends? Comparative Education Review,60(4), 644–672.

    Google Scholar 

  39. Komatsu, H., & Rappleye, J. (2017). A new global policy regime founded on invalid statistics? Hanushek, Woessmann, PISA, and economic growth. Comparative Education,53(2), 166–191.

    Google Scholar 

  40. Kreiner, S., & Christensen, K. B. (2014). Analyses of model fit and robustness. A new look at the PISA scaling model underlying ranking of countries according to reading literacy. Psychometrika. https://doi.org/10.1007/s11336-013-9347-z.

    Article  Google Scholar 

  41. Kuramoto, N., & Koizumi, R. (2016). Current issues in large-scale educational assessment in Japan: Focus on national assessment of academic ability and university entrance examinations. Assessment in Education: Principles, Policy & Practice. https://doi.org/10.1080/0969594X.2016.1225667.

    Article  Google Scholar 

  42. Labaree, D. (1997). Public goods, private goods: The American struggle over educational goals. American Educational Research Journal,34(1), 39–81.

    Google Scholar 

  43. Labaree, D. F. (2014). Let’s measure what no one teaches: PISA, NCLB, and the shrinking aims of education. Teachers College Record,116(9), 1–14.

    Google Scholar 

  44. Lewis, S. (2017). Policy, philanthropy and profit: The OECD’s PISA for schools and new modes of heterarchical educational governance. Comparative Education,53(4), 518–537. https://doi.org/10.1080/03050068.2017.1327246.

    Article  Google Scholar 

  45. Lingard, B., Rezai-Rashti, G., Martino, W., & Sellar, S. (2015). Globalizing educational accountabilities. London: Routledge.

    Google Scholar 

  46. Loveless, T. (2012). The 2012 brown center report on American education: How well are American students learning? Retrieved January 20, 2015 from Washington DC http://www.brookings.edu/~/media/Files/rc/reports/2012/0216_brown_education_loveless/0216_brown_education_loveless.pdf.

  47. Loveless, T. (2014). PISA’s China problem continues: A response to Schleicher, Zhang, and Tucker. Retrieved January 20, 2015 from http://www.brookings.edu/research/papers/2014/01/08-shanghai-pisa-loveless.

  48. Lundgren, U. P. (2011). PISA as a political instrument: One history behind the formulating of the PISA programme. In M. A. Pereyra, H.-G. Kotthoff, & R. Cowen (Eds.), Pisa under examination (pp. 17–30). New York: Springer.

    Google Scholar 

  49. Meyer, H.-D., et. al. (2014). OECD and Pisa tests are damaging education worldwideacademics. Retrieved Februrary 3, 2015 from https://www.theguardian.com/education/2014/may/06/oecd-pisa-tests-damaging-education-academics.

  50. Meyer, H.-D., & Benavot, A. (2013). PISA, power, and policy: The emergence of global educational governance. Oxford: Oxford University Press.

    Google Scholar 

  51. Morrison, H. (2013). Pisa 2012 major flaw exposed. Retrieved Februrary 3, 2015 from https://paceni.wordpress.com/2013/12/01/pisa-2012-major-flaw-exposed/.

  52. Ninomiya, S. (2016). The impact of PISA and the interrelation and development of assessment policy and assessment theory in Japan. Assessment in Education: Principles, Policy & Practice,26, 1–20.

    Google Scholar 

  53. Nyborg, H. (2007). Do recent large-scale cross-national student assessment studies neglect general intelligence g for political reasons? European Journal of Personality,21, 739–741.

    Google Scholar 

  54. OECD. (1999). Measuring student knowledge and skills: A new framework for assessment. Retrieved March 15, 2019 from Paris: http://www.oecd.org/education/school/programmeforinternationalstudentassessmentpisa/33693997.pdf.

  55. OECD. (2010). OECD programme for international student assessment. Retrieved March 15, 2019 from http://www.pisa.oecd.org/pages/0,2987,en_32252351_32235731_1_1_1_1_1,00.html.

  56. OECD. (2013). Ready to learn: Students engagement, drive, and self-beliefs. Retrieved March 15, 2019 from Paris: http://www.oecd.org/pisa/keyfindings/pisa-2012-results-volume-III.pdf.

  57. OECD. (2016a). PISA 2015 results (volume I): Excellence and equity in education. Retrieved March 15, 2019 from Paris: http://dx.doi.org/10.1787/9789264266490-en.

  58. OECD. (2016b). PISA 2015: Results in focus. PISA. Retrieved March 15, 2019 from https://www.oecd.org/pisa/pisa-2015-results-in-focus.pdf.

  59. OECD. (2017). PISA 2015 results: Students’ well-being. Retrieved March 15, 2019 from Paris http://www.keepeek.com/Digital-Asset-Management/oecd/education/pisa-2015-results-volume-iii_9789264273856-en-.Wk1WGrQ-fOQ#page1.

  60. OECD. (2018a). About PIAAC. Retrieved March 15, 2019 from http://www.oecd.org/skills/piaac/about/.

  61. OECD. (2018b). PISA for development. Retrieved March 15, 2019 from https://www.oecd.org/pisa/pisa-for-development/.

  62. OECD. (2018c). PISA for schools. Retrieved March 15, 2019 from http://www.oecd.org/PISA/pisa-for-schools/.

  63. OECD. (2018d). PISA: About. Retrieved March 15, 2019 from http://www.oecd.org/pisa/aboutpisa/.

  64. OECD. (nd). The international early learning and child well-being studyThe study. Retrieved March 15, 2019 from http://www.oecd.org/education/school/the-international-early-learning-and-child-well-being-study-the-study.htm.

  65. OECD/UNESCO. (2003). Literacy skills for the world of tomorrow—Further results from Pisa 2000. PISA. Retrieved October 23, 2017 from http://www.oecd.org/education/school/2960581.pdf.

  66. Pence, A. (2016). Baby PISA: Dangers that can arise when foundations shift. Journal of Childhood Studies,41(3), 54–58.

    Google Scholar 

  67. Pereyra, M. A., Kotthoff, H.-G., & Cowen, R. (2011). PISA under examination. New York: Springer.

    Google Scholar 

  68. Popkewitz, T. (2011). PISA: Numbers, standardizing conduct, and the alchemy of school subjects. In M. A. Pereyra, H.-G. Kotthoff, & R. Cowen (Eds.), Pisa under examination (pp. 31–46). New York: Springer.

    Google Scholar 

  69. Prais, S. J. (2003). Cautions on OECD’s recent educational survey (PISA). Oxford Review of Education,29, 139–163.

    Google Scholar 

  70. Rindermann, H. (2007). The g-factor of international cognitive ability comparisons: The homogeneity of results in PISA, TIMSS, PIRLS and IQ-tests across nations. European Journal of Personality,21, 667–706.

    Google Scholar 

  71. Rutkowski, L., & Rutkowski, D. (2016). A call for a more measured approach to reporting and interpreting PISA results. Educational Researcher,45(4), 252–257.

    Google Scholar 

  72. Sahlberg, P. (2011). Finnish lessons: What can the world learn from educational change in Finland?. New York: Teachers College Press.

    Google Scholar 

  73. Sahlberg, P. (2012). How GERM is infecting schools around the world. Retrieved October 23, 2017 from http://www.washingtonpost.com/blogs/answer-sheet/post/how-germ-is-infecting-schools-around-the-world/2012/06/29/gJQAVELZAW_blog.html-comments.

  74. Sahlberg, P. (2017). FinnishED leadership. Thousand Oaks: Corwin.

    Google Scholar 

  75. Schleicher, A. (2013). Are the Chinese cheating in PISA or are we cheating ourselves? Retrieved Februrary 3, 2015 from http://oecdeducationtoday.blogspot.com/2013/12/are-chinese-cheating-in-pisa-or-are-we.html.

  76. Schleicher, A. (2018). World class: How to build a 21st-century school system. Paris: OECD.

    Google Scholar 

  77. Schuelka, M. J. (2013). Excluding students with disabilities from the culture of achievement: The case of the TIMSS, PIRLS, and PISA. Journal of Education Policy,28(2), 216–230.

    Google Scholar 

  78. Sellar, S., & Lingard, B. (2014). The OECD and the expansion of PISA: New global modes of governance in education. British Educational Research Journal,40(6), 917–936.

    Google Scholar 

  79. Shirley, D. (2017). The new imperatives of educational change: Achievement with integrity. New York: Routledge.

    Google Scholar 

  80. Sjøberg, S. (2012). PISA: Politics, fundamental problems and intriguing results. Recherches en Education,14, 1–21.

    Google Scholar 

  81. Sjøberg, S. (2015a). OECD, PISA, and globalization: The influence of the international assessment regime. In C. H. Tienken & C. A. Mullen (Eds.), Education policy perils (pp. 114–145). London: Routledge.

    Google Scholar 

  82. Sjøberg, S. (2015b). PISA and global educational governance—A critique of the project, its uses and implications. Eurasia Journal of Mathematics, Science and Technology Education,11(1), 111–127.

    Google Scholar 

  83. Solheim, O. J., & Lundetræ, K. (2018). Can test construction account for varying gender differences in international reading achievement tests of children, adolescents and young adults? A study based on Nordic results in PIRLS, PISA and PIAAC. Assessment in Education: Principles, Policy and Practice,25(1), 107–126.

    Google Scholar 

  84. Stewart, W. (2013). Is Pisa fundamentally flawed? Retrieved Februrary 3, 2015 from http://www.tes.co.uk/article.aspx?storycode=6344672.

  85. Stromquist, N. P. (2016). Using regression analysis to predict countries’ economic growth: Illusion and fact in education policy. Real-World Economics Review,76, 65–74.

    Google Scholar 

  86. Tröhler, D. (2013). The OECD and cold war culture: Thinking historically about PISA. In PISA, power, and policy: The emergence of global educational governance (pp. 141–161).

  87. Tucker, M. (Ed.). (2011). Surpassing Shanghai: An agenda for American education built on the world’s leading systems. Boston: Harvard Education Press.

    Google Scholar 

  88. Tucker, M. (2014). Chinese lessons: Shanghai’s rise to the top of the PISA league tables. Retrieved October 23, 2017 from Washington DC. http://www.ncee.org/wp-content/uploads/2013/10/ChineseLessonsWeb.pdf.

  89. Tucker, M. (2016). Asian countries take the U.S. to school. The Atlantic. Retrieved October 23, 2017 from https://www.theatlantic.com/education/archive/2016/02/us-asia-education-differences/471564/.

  90. Uljens, M. (2007). The hidden curriculum of PISA: The promotion of neo-liberal policy by educational assessment. In S. T. Hopmann, G. Brinek, & M. Retzl (Eds.), PISA zufolge PISA—PISA According to PISA (pp. 295–303). Berlin: Lit Verlag.

    Google Scholar 

  91. Urban, M. (2017). We need meaningful, systemic evaluation, not a preschool PISA. Global Education Review,4(2), 18–24.

    Google Scholar 

  92. Wu, M. (2009). A comparison of PISA and TIMSS 2003 achievement results in mathematics. Prospects,39(1), 33.

    Google Scholar 

  93. Zhao, Y. (2009). Catching up or leading the way: American education in the age of globalization. Alexandria: ASCD.

    Google Scholar 

  94. Zhao, Y. (2012). World class learners: Educating creative and entrepreneurial students. Thousand Oaks: Corwin.

    Google Scholar 

  95. Zhao, Y. (2014). Who’s afraid of the big bad dragon: Why China has the best (and Worst) education system in the world. San Francisco: Jossey-Bass.

    Google Scholar 

  96. Zhao, Y. (2015). Lessons that matter: What we should learn from Asian school systems. Retrieved July 20, 2017 from Melbourne. http://www.mitchellinstitute.org.au/reports/lessons-that-matter-what-should-we-learn-from-asias-school-systems/.

  97. Zhao, Y. (2016a). From deficiency to strength: Shifting the mindset about education inequality. Journal of Social Issues,72(4), 716–735.

    Google Scholar 

  98. Zhao, Y. (2016b). Who’s afraid of PISA: The fallacy of international assessments of system performance. In A. Harris & M. S. Jones (Eds.), Leading futures (pp. 7–21). Thousand Oaks: Sage.

    Google Scholar 

  99. Zhao, Y. (2017a). Fatal attraction: Why the west must stop copying China’s flawed education system. New Internationalist,505, 24–25.

    Google Scholar 

  100. Zhao, Y. (2017b). What works can hurt: Side effects in education. Journal of Educational Change,18(1), 1–19.

    Google Scholar 

  101. Zhao, Y. (2018a). The changing context of teaching and implications for teacher education. Peabody Journal of Education,93, 1–14.

    Google Scholar 

  102. Zhao, Y. (2018b). Shifting the education paradigm: Why international borrowing is no longer sufficient for improving education in China. ECNU Review of Education,1(1), 76–106.

    Google Scholar 

  103. Zhao, Y. (2018c). What works may hurt: Side effects in education. New York: Teachers College Press.

    Google Scholar 

  104. Zhao, Y., & Gearin, B. (2016). Squeezed out. In D. Ambrose & R. J. Sternberg (Eds.), Creative intelligence in the 21st century (pp. 121–138). Berlin: Springer.

    Google Scholar 

  105. Zhao, Y., & Wang, Y. (2018). Guarding the past or inventing the future: Education reforms in East Asia. In Y. Zhao & B. Gearin (Eds.), Imagining the future of global education: Dreams and nightmares (pp. 143–159). New York: Routledge.

    Google Scholar 

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Yong Zhao.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Zhao, Y. Two decades of havoc: A synthesis of criticism against PISA. J Educ Change 21, 245–266 (2020). https://doi.org/10.1007/s10833-019-09367-x

Download citation

Keywords

  • PISA
  • International assessment
  • Educational policy
  • International and comparative education
  • Purpose of education
  • Future of education
  • Education reform
  • Educational change