Secondary Students’ Understanding of Ecosystems: a Learning Progression Approach

  • Hui JinEmail author
  • Hyo Jeong Shin
  • Hayat Hokayem
  • Farah Qureshi
  • Thomas Jenkins


This study describes how we developed a learning progression (LP) for systems thinking in ecosystems and collected preliminary validity evidence for the LP. In particular, the LP focuses on how middle and high school students use discipline-specific systems thinking concepts (e.g. feedback loops and energy pyramid) to analyze and explain the interdependent relationships in ecosystems and humans’ impact on those relationships. We administrated written assessments with 596 secondary students. Based on the data, we developed and validated an LP for systems thinking in ecosystems. The LP contains four levels that describe increasingly sophisticated reasoning patterns that students commonly use to explain phenomena about interdependent relationships in ecosystems. We used a Wright Map based on the Rasch model for polyotmous data to evaluate the validity of the LP. We also used the LP to compare the performance of students from different subgroups in terms of socioeconomic status (SES), gender, and school settings. The data suggests performance gaps for students with low SES and students from urban schools, but not for other traditionally under-served or under-represented groups such as female students and students from rural schools.


Ecosystems Learning progression Performance gap Systems thinking 



This work was supported by Education Testing Service CBAL (Cognitively Based Assessment of, for, and as Learning) Initiative. The authors wish to thank Randy Bennett for his valuable support throughout the research.

Supplementary material

10763_2017_9864_MOESM1_ESM.docx (1.3 mb)
ESM 1 (DOCX 1320 kb)


  1. Alonzo, A. C., & Steedle, J. T. (2009). Developing and assessing a force and motion learning progression. Science Education, 93, 389–421.CrossRefGoogle Scholar
  2. Ben-Zvi Assaraf, O., & Orion, N. (2005). Development of system thinking skills in the context of earth systems education. Journal of Research in Science Teaching, 42, 400–419.Google Scholar
  3. Ben-Zvi Assaraf, O., & Orion, N. (2010). Systems thinking skills at the elementary school level. Journal of Research in Science Teaching, 5, 540–563.Google Scholar
  4. Berland, L. K., & McNeill, K. L. (2010). A learning progression for scientific argumentation: Understanding student work and designing supportive instructional contexts. Science Education, 94(5), 765–793.CrossRefGoogle Scholar
  5. Camilli, G., & Shepard, L. A. (1994). Methods for identifying biased test items. Thousand Oaks, CA: SAGE Publications.Google Scholar
  6. Chi, M. T. H., Roscoe, R. D., Slotta, J. D., Roy, M., & Chase, C. C. (2012). Misconceived causal explanations for emergent processes. Cognitive Science, 36(1), 1–61.CrossRefGoogle Scholar
  7. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, 37–46.CrossRefGoogle Scholar
  8. Cohen, J. (1968). Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological Bulletin, 70, 213–220.CrossRefGoogle Scholar
  9. Dauer, J. T., Momsen, J. L., Speth, E. B., Makohon-Moore, S. C., & Long, T. M. (2013). Analyzing change in students’ gene-to-evolution models in college-level introductory biology. Journal of Research in Science Teaching, 50, 639–659.CrossRefGoogle Scholar
  10. Duncan, R. G., Rogat, A. D., & Yarden, A. (2009). A learning progression for deepening students’ understanding of modern genetics across the 5th–10th grades. Journal of Research in Science Teaching, 46, 655–674.CrossRefGoogle Scholar
  11. Forbes, C. T., Zangori, L., & Schwartz, C. V. (2015). Empirical validation of integrated learning performances for hydrologic phenomena: 3rd-grade students’ model-driven explanation-construction. Journal of Research in Science Teaching, 52, 895–921.CrossRefGoogle Scholar
  12. Furtak, E. M. (2012). Linking a learning progression for natural selection to teachers’ enactment of formative assessment. Journal of Research in Science Teaching, 49, 1181–1210.CrossRefGoogle Scholar
  13. Furtak, E. M., & Heredia, S. C. (2014). Exploring the influence of learning progressions in two teacher communities. Journal of Research in Science Teaching, 51, 982–1020.CrossRefGoogle Scholar
  14. Gotwals, A. W., & Songer, N. B. (2013). Validity evidence for learning progression-based assessment items that fuse core disciplinary ideas and science practices. Journal of Research in Science Teaching, 50(5), 597–626.Google Scholar
  15. Gunckel, K., Covitt, B., Salinas, I., & Anderson, C. W. (2012). A learning progression for water in socio-ecological systems. Journal of Research in Science Teaching, 49(7), 843–868.CrossRefGoogle Scholar
  16. Hadenfeldt, J. C., Neumann, K., Bernholt, S., Liu, X., & Parchmann, I. (2016). Students’ progression in understanding the matter concept. Journal of Research in Science Teaching, 53, 683–708.CrossRefGoogle Scholar
  17. Hmelo-Silver, C. E., Marathe, S., & Liu, L. (2007). Fish swim, rocks sit, and lungs breathe: Expert-novice understanding of complex systems. Journal of the Learning Sciences, 16, 307–331.CrossRefGoogle Scholar
  18. Hmelo-Silver, C. E., & Pfeffer, M. G. (2004). Comparing expert and novice understanding of a complex system from the perspective of structures, behaviors, and functions. Cognitive Science, 28, 127–138.CrossRefGoogle Scholar
  19. Hogan, K. (2000). Assessing students’ systems reasoning in ecology. Journal of Biological Education, 35, 22–28.CrossRefGoogle Scholar
  20. Hokayem, H., & Gotwals, A. W. (2016). Early elementary students’ understanding of complex ecosystems: A learning progression approach. Journal of Research in Science Teaching, 53(10), 1524–1545.Google Scholar
  21. Hokayem, H., Ma, J., & Jin, H. (2015). A learning progression for feedback loop reasoning at lower elementary level. Journal of Biological Education. 49(3), 246–260.Google Scholar
  22. Holland, W., & Wainer, H. (1993). Differential item functioning. New York, NY: Routledge.Google Scholar
  23. Jin, H., & Anderson, C. W. (2012a). Development of assessments for a learning progression on carbon cycling in socio-ecological systems. In A. Alonzo & A. W. Gotwals (Eds.), Learning progressions in science: Current challenges and future directions (pp. 151–182). Rotterdam, The Netherlands: Sense Publishers.Google Scholar
  24. Jin, H., & Anderson, C. W. (2012b) A learning progression for energy in socio-ecological systems. Journal of Research in Science Teaching, 49(9), 1149–1180.Google Scholar
  25. Jin, H., Johnson, M. E., Shin, H.-J., & Anderson, C. W. (2017). Promoting student progressions in science classrooms: A video study. Journal of Research in Science Teaching54(7), 852–883.Google Scholar
  26. Jin, H., Mikeska, J., Hokayem, H., & Mavronikolas, E. (2017, April). Learning progression research: Toward the coherence in teaching and learning of science. Paper presented at the annual conference of the National Association for Research in Science Teaching (NARST). San Antonio, TX: NARST.Google Scholar
  27. Jin, H., Shin, H.-J., Johnson, E. M., Kim, J., & Anderson, C. W. (2015). Developing learning progression-based teacher knowledge measures. Journal of Research in Science Teaching, 52(9), 1269–1295.Google Scholar
  28. Jin, H., & Wei, X. (2014). Using ideas from the history of science and linguistics to develop a learning progression for energy in socio-ecological systems. In R. F. Chen, A. Eisenkraft, F. Fortus, J. Krajcik, K. Neumann, J. C. Nordine, & A. Scheff (Eds.), Teaching and learning of energy in K-12 Education (pp. 157–174). New York, NY: Springer.Google Scholar
  29. Jin, H., Zhan, L., & Anderson, C. W. (2013). Developing a fine-grained learning progression framework for carbon-transforming processes. International Journal of Science Education, 35(10), 1663–1697.Google Scholar
  30. Johnson, P., & Tymms, P. (2011). The emergence of a learning progression in middle school chemistry. Journal of Research in Science Teaching, 48, 849–877.CrossRefGoogle Scholar
  31. Kane, M. T. (2006). Validation. In R. L. Brennan (Ed.), Educational measurement (4th ed., pp. 17–64). New York, NY: American Council on Education, Macmillan.Google Scholar
  32. Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159–174.CrossRefGoogle Scholar
  33. Lee, O., & Luykx, A. (2007). Science education and student diversity: Race/ethnicity, language, culture, and socioeconomic status. In S. K. Abell & N. G. Lederman (Eds.), Handbook of research on science education (pp. 171–198). New York, NY: Routledge.Google Scholar
  34. Masters, G. N. (1982). A Rasch model for partial credit scoring. Psychometrika, 47(2), 149–174.CrossRefGoogle Scholar
  35. Mohan, L., Chen, J., & Anderson, C. W. (2009). Developing a multi-year learning progression for carbon cycling in socio-ecological systems. Journal of Research in Science Teaching, 46, 675–698.CrossRefGoogle Scholar
  36. Neumann, K., Viering, T., Boone, W., & Fischer, H. E. (2013). Towards a learning progression of energy. Journal of Research in Science Teaching, 50(2), 162–188.CrossRefGoogle Scholar
  37. NGSS Lead States (2013). Next generation science standards: For states, by states. Washington, D.C.: Achieve, Inc..Google Scholar
  38. National Research Council (2007). Taking science to school: Learning and teaching science in grade K-8. Washington, D.C.: The National Academies Press.Google Scholar
  39. National Research Council (2012). A framework for K-12 science education: Practices, crosscutting concepts, and core ideas. Washington, D.C.: The National Academies Press.Google Scholar
  40. Plummer, J. D., & Krajcik, J. (2010). Building a learning progression for celestial motion: Elementary levels from an earth-based perspective. Journal of Research in Science Teaching, 47, 76–787.CrossRefGoogle Scholar
  41. Plummer, J. D., & Maynard, L. (2014). Building a learning progression for celestial motion: An exploration of students’ reasoning about the seasons. Journal of Research in Science Teaching, 51, 902–929.CrossRefGoogle Scholar
  42. Rasch, G. (1960). Studies in mathematical psychology: I. Probabilistic models for some intelligence and attainment tests. Oxford, England: Nielsen & Lydiche.Google Scholar
  43. Resnick, M. (1996). Beyond the centralized mindset. The Journal of the Learning Sciences, 5(1), 1–22.CrossRefGoogle Scholar
  44. Resnick, M., & Wilensky, U. (1998). Diving into complexity: Developing probabilistic decentralized thinking through role-playing activities. The Journal of the Learning Sciences, 7(2), 153–172.CrossRefGoogle Scholar
  45. Ripple, W. J., Larsen, E. J., Renkin, R. A., & Smith, D. W. (2001). Trophic cascades among wolves, elk, and aspen on Yellowstone National Park’s northern range. Biological Conservation, 102, 227–334.CrossRefGoogle Scholar
  46. Robitzsch, A., Kiefer, T., & Wu, M. (2017). TAM: Test analysis modules. R package version 1.99992–0. Retrieved from
  47. Schmidt, W. H., McKnight, C. C., & Raizen, S. A. (1997). A splintered vision: An investigation of U.S. science and mathematics education. Boston, MA: Kluwer.Google Scholar
  48. Schwarz, C. V., Reiser, B., Davis, E. A., Kenyon, L., Acher, A., Fortus, D., Krajcik, J. (2009). Developing a learning progression for scientific modeling: Making scientific modeling accessible and meaningful for learners. Journal of Research in Science Teaching, 46(6), 632–654.Google Scholar
  49. Schwarz, G. (1978). Estimating the dimension of a model. The Annals of Statistics, 6, 461–464.CrossRefGoogle Scholar
  50. Smith, C. L., Wiser, M., Anderson, C. W., & Krajcik, J. (2006). Implications of research on children’s learning for standards and assessment: A proposed learning progression for matter and the atomic molecular theory. Measurement: Interdisciplinary Research and Perspectives, 14(1–2), 1–98.Google Scholar
  51. Sudweeks, R. R., & Tolman, R. R. (1993). Empirical versus subjective procedures for identifying gender differences in science test items. Journal of Research in Science Teaching, 30(1), 3–19.CrossRefGoogle Scholar
  52. Sweeney, L. B., & Sterman, J. D. (2000). Bathtub dynamics: Initial results of a systems thinking inventory. System Dynamics Review, 16, 249–286.CrossRefGoogle Scholar
  53. Sweeney, L. B., & Sterman, J. D. (2007). Thinking about systems: Student and teacher conceptions of natural and social systems. System Dynamics Review, 23, 285–312.CrossRefGoogle Scholar
  54. Wilensky, U., & Resnick, M. (1999). Thinking in levels: A dynamic systems approach to making sense of the world. Journal of Science Education and Technology, 8, 3–19.CrossRefGoogle Scholar
  55. Wilson, M. (2009). Measuring progressions: Assessment structures underlying a learning progression. Journal of Research in Science Teaching, 46, 716–730.CrossRefGoogle Scholar

Copyright information

© Ministry of Science and Technology, Taiwan 2017

Authors and Affiliations

  1. 1.Student and Teacher Research Center, Educational Testing ServicePrincetonUSA
  2. 2.Research and Development Division, Educational Testing ServicePrincetonUSA
  3. 3.Andrews Institute of Mathematics & Science EducationTexas Christian UniversityFort WorthUSA
  4. 4.National Assessment of Educational Progress, Educational Testing ServicePrincetonUSA
  5. 5.Assessment Development, Educational Testing ServiceSan AntonioUSA

Personalised recommendations