Skip to main content
Log in

Development and Validation of a Reading in Science Holistic Assessment (RISHA): a Rasch Measurement Study

  • Published:
International Journal of Science and Mathematics Education Aims and scope Submit manuscript

Abstract

Researchers in science education lacks valid and reliable instruments to assess students’ disciplinary and epistemic reading of scientific texts. The main purpose of this study was to develop and validate a Reading in Science Holistic Assessment (RISHA) to assess students’ holistic reading of scientific texts. RISHA measures students’ content, procedural, and epistemic domains of reading two texts, one history-of-science text and another socio-scientific text. The initial 24-item RISHA was administered to 161 Grade 9 students from 3 schools. The multidimensional Rasch partial credit model was used to analyze the reliability and validity of RISHA. All items demonstrated good fit and reliability. According to logit scores generated for each domain in Rasch analysis, students in our study performed better in content domain and less well in the epistemic domain. Students also performed significantly better in the epistemic domain of the socio-scientific text than in the history-of-science text. RISHA provides accurate measures in various domains of reading scientific texts and various contexts of scientific texts. We propose that RISHA could potentially be applied to studying the effect of reading-science intervention or predictors of students’ performance in each domain of reading scientific texts.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  • Abrahams, I., Reiss, M. J., & Sharpe, R. M. (2013). The assessment of practical work in school science. Studies in Science Education, 49(2), 209–251.

    Article  Google Scholar 

  • Avsar Erumit, B., & Yuksel, T. (2023). Developing and using physical dynamic models on socioscientific issues to present nature of science ideas. International Journal of Science and Mathematics Education, 21(4), 1031–1056.

    Article  Google Scholar 

  • Bernholt, S., Härtig, H., & Retelsdorf, J. (2022). Reproduction rather than comprehension? Analysis of gains in students’ science text comprehension. Research in Science Education, 53(3), 493–506. https://doi.org/10.1007/s11165-022-10066-6

    Article  Google Scholar 

  • Boggs, G. L., Wilson, N. S., Ackland, R. T., Danna, S., & Grant, K. B. (2016). Beyond the Lorax: Examining children’s books on climate change. The Reading Teacher, 69(6), 665–675.

    Article  Google Scholar 

  • Bond, T., Yan, Z., & Heene, M. (2020). Applying the Rasch model: Fundamental measurement in the human sciences. Routledge.

  • Boone, W. J., Staver, J. R., & Yale, M. S. (2013). Rasch analysis in the human sciences. Springer.

  • Chan, H. Y., Cheung, K. K. C., & Erduran, S. (2023). Science communication in the media and human mobility during the COVID-19 pandemic: A time series and content analysis. Public Health, 218, 106–113.

    Article  Google Scholar 

  • Chen, S.-Y., Chen, C.-H., & Liu, S.-Y. (2022). History of science reading materials as everyday homework to improve middle school students’ epistemological beliefs about science. International Journal of Science and Mathematics Education, 20(Suppl. 1), S69–S92. https://doi.org/10.1007/s10763-022-10285-3

    Article  Google Scholar 

  • Cheung, C. M. (2017). “Science (S1–3) – Learning and teaching resources” (Set C). Retrieved from https://cd1.edb.hkedcity.net/cd/science/multimedia/media1_1_e.htm

  • Cheung, K. K. C., & Winterbottom, M. (2021). Exploring students’ visualisation competence with photomicrographs of villi. International Journal of Science Education, 43(14), 2290–2315.

    Article  Google Scholar 

  • Cheung, K. K. C., Chan, H. Y., & Erduran, S. (2023). Communicating science in the COVID-19 news in the UK during Omicron waves: Exploring representations of nature of science with epistemic network analysis. Humanities and Social Sciences Communications, 10(1), 1–14.

    Article  Google Scholar 

  • Cheung, K. K. C., & Pun, J. K. (2023). The use of epistemic network analysis in analysing classroom discourse in EMI-science classrooms. In S. M. Curle & J. K. H. Pun (Eds.), Qualitative research methods in English medium instruction for emerging researchers (pp. 33–44). Routledge.

    Chapter  Google Scholar 

  • Cheung, K. K. C., & Sonkqayi, G. (2023). Students’ science achievement in cognitive domains: Effects of practical work and clarity of instruction. Research in Science & Technological Education, 1–18.

  • Cheung, K. K. C., & Winterbottom, M. (2023). Students’ integration of textbook representations into their understanding of photomicrographs: Epistemic network analysis. Research in Science & Technological Education, 41(2), 544–563.

    Article  Google Scholar 

  • Clough, M. P. (2006). Learners’ responses to the demands of conceptual change: Considerations for effective nature of science instruction. Science & Education, 15, 463–494.

    Article  Google Scholar 

  • Conley, A. M., Pintrich, P. R., Vekiri, I., & Harrison, D. (2004). Changes in epistemological beliefs in elementary science students. Contemporary Educational Psychology, 29(2), 186–204. https://doi.org/10.1016/j.cedpsych.2004.01.004

    Article  Google Scholar 

  • Curriculum Development Council and Hong Kong Examinations and Assessment Authority (2017). Science education: Key learning area curriculum guide (primary 1—secondary 6). Hong Kong: Curriculum Development Council.

  • Dagenais, A. (2010). Teaching high school physics with a story-line. Interchange, 41(4), 335–345. https://doi.org/10.1007/s10780-010-9134-z

    Article  Google Scholar 

  • Dass, P. M. (2005). Understanding the nature of scientific enterprise (NOSE) through a discourse with its history: The influence of an undergraduate ‘history of science’course. International Journal of Science and Mathematics Education, 3, 87–115.

    Article  Google Scholar 

  • Fang, Z. (2006). The language demands of science reading in middle school. International Journal of Science Education, 28(5), 491–520. https://doi.org/10.1080/09500690500339092

    Article  Google Scholar 

  • Fang, Z. (2008). Going beyond the fab five: Helping students cope with the unique linguistic challenges of expository reading in intermediate grades. Journal of Adolescent & Adult Literacy, 51(6), 476–487.

    Article  Google Scholar 

  • Farber, D. A. (2007). Adapting to climate change: Who should pay. J. Land Use & Envtl. l., 23, 1.

    Google Scholar 

  • Fazio, X., & Gallagher, T. L. (2019). Science and language integration in elementary classrooms: Instructional enactments and student learning outcomes. Research in Science Education, 49, 959–976.

    Article  Google Scholar 

  • Fazio, X., Gallagher, T. L., & DeKlerk, C. (2022). Exploring adolescents’ critical reading of socioscientific topics using multimodal texts. International Journal of Science and Mathematics Education, 20(Suppl. 1), S93–S116.

    Article  Google Scholar 

  • Flesch, R. (2007). Flesch-Kincaid readability test. Retrieved October, 26(3), 2007.

  • Ford, M. J., & Wargo, B. M. (2012). Dialogic framing of scientific content for conceptual and epistemic understanding. Science Education, 96(3), 369–391.

    Article  Google Scholar 

  • Gardiner, S. M. (2010). Ethics and climate change: An introduction. Wiley Interdisciplinary Reviews: Climate Change, 1(1), 54–66.

    Google Scholar 

  • Grigg, K., & Manderson, L. (2016). The Australian racism, acceptance, and cultural-ethnocentrism scale (RACES): Item response theory findings. International Journal for Equity in Health, 15, 1–16.

    Article  Google Scholar 

  • Hambleton, R. K., & Jones, R. W. (1993). Comparison of classical test theory and item response theory and their applications to test development. Educational Measurement: Issues and Practice, 12(3), 38–47.

    Article  Google Scholar 

  • Härtig, H., Bernholt, S., Fraser, N., Cromley, J. G., & Retelsdorf, J. (2022). Comparing reading comprehension of narrative and expository texts based on the direct and inferential mediation model. International Journal of Science and Mathematics Education, 20(1), 17–41. https://doi.org/10.1007/s10763-022-10302-5

    Article  Google Scholar 

  • Holliday, W. G., Yore, L. D., & Alvermann, D. E. (1994). The reading–science learning–writing connection: Breakthroughs, barriers, and promises. Journal of Research in Science Teaching, 31(9), 877–893.

    Article  Google Scholar 

  • Jian, Y.-C. (2018). Teaching fourth-grade students of different reading abilities to read biological illustrations and integrate in-text information: An empirical experiment. Research in Science Education, 50(6), 2269–2282. https://doi.org/10.1007/s11165-018-9778-8

    Article  Google Scholar 

  • Khishfe, R. (2023). Improving students’ conceptions of nature of science: A review of the literature. Science & Education, 32, 1887–1931. https://doi.org/10.1007/s11191-022-00390-8

    Article  Google Scholar 

  • Khishfe, R., Alshaya, F. S., BouJaoude, S., Mansour, N., & Alrudiyan, K. I. (2017). Students’ understandings of nature of science and their arguments in the context of four socio-scientific issues. International Journal of Science Education, 39(3), 299–334.

    Article  Google Scholar 

  • Kim, S. Y., & Irving, K. E. (2010). History of science as an instructional context: Student learning in genetics and nature of science. Science & Education, 19, 187–215.

    Article  Google Scholar 

  • Klopfer, L. E., & Aikenhead, G. S. (2022). Humanistic science education: The history of science and other relevant contexts. Science Education, 106(3), 490–504.

    Article  Google Scholar 

  • Kolstø, S. D. (2008). Science education for democratic citizenship through the use of the history of science. Science & Education, 17, 977–997.

    Article  Google Scholar 

  • Lammers, A., Goedhart, M. J., & Avraamidou, L. (2019). Reading and synthesising science texts using a scientific argumentation model by undergraduate biology students. International Journal of Science Education, 41(16), 2323–2346.

    Article  Google Scholar 

  • Lederman, N. G. (2013). Nature of science: Past, present, and future. In S. K. Abell & N. G. Lederman (Eds.), Handbook of research on science education (pp. 831–879). Routledge.

    Google Scholar 

  • Lederman, N. G., Abd-El-Khalick, F., Bell, R. L., & Schwartz, R. S. (2002). Views of nature of science questionnaire: Toward valid and meaningful assessment of learners’ conceptions of nature of science. Journal of Research in Science Teaching, 39(6), 497–521.

    Article  Google Scholar 

  • Leu, D. J., Kinzer, C. K., Coiro, J., Castek, J., & Henry, L. A. (2017). New literacies: A dual-level theory of the changing nature of literacy, instruction, and assessment. Journal of Education, 197(2), 1–18.

    Article  Google Scholar 

  • Leung, J. S. C. (2020). Students’ adherences to epistemic understanding in evaluating scientific claims. Science Education, 104(2), 164–192.

    Article  Google Scholar 

  • Li, C., Yu, J., & Li, G. (2023). Development of the representation of the nature of science in textbooks: The case of high school biology textbooks in mainland China. International Journal of Science and Mathematics Education, 21(6), 1749–1768.

    Article  Google Scholar 

  • Lin, C. Y., Cheng, J. H., & Chang, W. H. (2010). Making science vivid: Using a historical episodes map. International Journal of Science Education, 32(18), 2521–2531.

    Article  Google Scholar 

  • Lindsey, R., & Dahlman, L. (2023). Climate change: Global temperature. Retrieved from https://www.climate.gov/news-features/understanding-climate/climate-change-global-temperature#:~:text=Earth's%20temperature%20has%20risen%20by,based%20on%20NOAA's%20temperature%20data.

  • Masters, G. N. (1982). A rasch model for partial credit scoring. Psychometrika, 47(2), 149–174. https://doi.org/10.1007/BF02296272

    Article  Google Scholar 

  • Matkins, J. J., & Bell, R. L. (2007). Awakening the scientist inside: Global climate change and the nature of science in an elementary science methods course. Journal of Science Teacher Education, 18(2), 137–163.

    Article  Google Scholar 

  • Matthews, M. R. (2014). Science teaching: The contribution of history and philosophy of science. Routledge.

  • Mccomas, W. F. (2011). The history of science and the future of science education: A typology of approaches to history of science in science instruction. In P. V. Kokkotas, K. S. Malamitsa, & A. A. Rizaki (Eds.), Adapting historical knowledge production to the classroom (pp. 37–53). Brill.

    Chapter  Google Scholar 

  • Millar, R., Lubben, F., Got, R., & Duggan, S. (1994). Investigating in the school science laboratory: Conceptual and procedural knowledge and their influence on performance. Research Papers in Education, 9(2), 207–248.

    Article  Google Scholar 

  • Monk, M., & Osborne, J. (1997). Placing the history and philosophy of science on the curriculum: A model for the development of pedagogy. Science Education, 81(4), 405–424.

    Article  Google Scholar 

  • National Research Council. (2012). A framework for K-12 science education: Practices, crosscutting concepts, and core ideas. National Academies Press.

  • Nigro, R. G., & Trivelato, S. F. (2012). Knowledge, its application, and attitudes associated with the reading of diverse genres of science texts. International Journal of Science Education, 34(16), 2529–2564.

    Article  Google Scholar 

  • Norris, S. P., & Phillips, L. M. (2003). How literacy in its fundamental sense is central to scientific literacy. Science Education, 87(2), 224–240.

    Article  Google Scholar 

  • Norris, S. P., & Phillips, L. M. (2008). Reading as inquiry. In R. A. Duschl & R. E. Grandy (Eds.), Teaching scientific inquiry (pp. 233–262). Brill.

    Chapter  Google Scholar 

  • Organization for Economic Co-operation and Development. (2019). PISA 2018 Assessment and Analytical Framework. Author.

  • Oliveras, B., Márquez, C., & Sanmartí, N. (2013). The use of newspaper articles as a tool to develop critical thinking in science classes. International Journal of Science Education, 35(6), 885–905. https://doi.org/10.1080/09500693.2011.586736

    Article  Google Scholar 

  • Ozuru, Y., Dempsey, K., & McNamara, D. S. (2009). Prior knowledge, reading skill, and text cohesion in the comprehension of science texts. Learning and Instruction, 19(3), 228–242. https://doi.org/10.1016/j.learninstruc.2008.04.003

    Article  Google Scholar 

  • Perfetti, C. A., Landi, N., & Oakhill, J. (2005). The acquisition of reading comprehension skill. In M. J. Snowling & C. Hulme (Eds.), The science of reading: A handbook (pp. 227–247). Blackwell Publishing.

    Chapter  Google Scholar 

  • Pun, J. K., Fu, X., & Cheung, K. K. C. (2023). Language challenges and coping strategies in English Medium Instruction (EMI) science classrooms: A critical review of literature. Studies in Science Education, 1–32.

  • Quellmalz, E., & Hoskyn, J. (1996). Classroom assessment of reasoning strategies. In G. D. Phye (Ed.), Handbook of classroom assessment (pp. 103–130). Elsevier.

    Chapter  Google Scholar 

  • Rauch, D. P., & Hartig, J. (2010). Multiple-choice versus open-ended response formats of reading test items: A two-dimensional IRT analysis. Psychological Test and Assessment Modeling, 52(4), 354–379.

    Google Scholar 

  • Rasch, G. (1966). An item analysis which takes individual differences into account. Br J Math Stat Psychol, 19(1), 49–57. https://doi.org/10.1111/j.2044-8317.1966.tb00354.x

    Article  Google Scholar 

  • Sadler, T. D., Chambers, F. W., & Zeidler, D. L. (2004). Student conceptualizations of the nature of science in response to a socioscientific issue. International Journal of Science Education, 26(4), 387–409.

    Article  Google Scholar 

  • Sadler, T. D., & Dawson, V. (2012). Socio-scientific issues in science education: Contexts for the promotion of key learning outcomes. In B. J. Fraser, K. G. Tobin, & C. J. McRobbie (Eds.), Second international handbook of science education (pp. 799–809). Springer.

    Chapter  Google Scholar 

  • Sbeglia, G. C., & Nehm, R. H. (2019). Do you see what I-SEA? A Rasch analysis of the psychometric properties of the Inventory of Student Evolution Acceptance. Science Education, 103(2), 287–316.

    Article  Google Scholar 

  • Secko, D. M., Amend, E., & Friday, T. (2013). Four models of science journalism: A synthesis and practical assessment. Journalism Practice, 7(1), 62–80.

    Article  Google Scholar 

  • Shepardson, D. P., & Gummer, E. S. (2001). A framework for thinking about and planning classroom assessments in science. In D. P. Shepardson (Ed.), Assessment in science: A guide to professional development and classroom practice (pp. 83–97). Springer.

    Chapter  Google Scholar 

  • Shymansky, J. A., Yore, L. D., & Good, R. (1991). Elementary school teachers’ beliefs about and perceptions of elementary school science, science reading, science textbooks, and supportive instructional factors. Journal of Research in Science Teaching, 28(5), 437–454.

    Article  Google Scholar 

  • Smith, E. A., & Senter, R. J. (1967). Automated readability index (Vol.66, No.220). Aerospace Medical Research Laboratories, Aerospace Medical Division, Air Force Systems Command.

  • Stang Lund, E., Bråten, I., Brandmo, C., Brante, E. W., & Strømsø, H. I. (2019). Direct and indirect effects of textual and individual factors on source-content integration when reading about a socio-scientific issue. Reading and Writing, 32, 335–356.

    Article  Google Scholar 

  • Stang Lund, E., Bråten, I., Brante, E. W., & Strømsø, H. I. (2017). Memory for textual conflicts predicts sourcing when adolescents read multiple expository texts. Reading Psychology, 38(4), 417–437.

    Article  Google Scholar 

  • Strømsø, H. I., & Bråten, I. (2009). Beliefs about knowledge and knowing and multiple-text comprehension among upper secondary students. Educational Psychology, 29(4), 425–445.

    Article  Google Scholar 

  • Strømsø, H. I., Bråten, I., & Samuelstuen, M. S. (2008). Dimensions of topic-specific epistemological beliefs as predictors of multiple text understanding. Learning and Instruction, 18(6), 513–527.

    Article  Google Scholar 

  • Symons, C. (2017). Supporting emergent bilinguals’ argumentation: Evaluating evidence in informational science texts. Linguistics and Education, 38, 79–91.

    Article  Google Scholar 

  • Tang, K.-S. (2021). The interconnections among metadiscourse, metalanguage, and metacognition: Manifestation and application in classroom discourse. Linguistics and Education, 65, 100977.

    Article  Google Scholar 

  • Tang, K.-S., & Rappa, N. A. (2021). The role of metalanguage in an explicit literacy instruction on scientific explanation. International Journal of Science and Mathematics Education, 19(7), 1311–1331.

    Article  Google Scholar 

  • Tang, K. S., Lin, S.-W., & Kaur, B. (2022). Mapping and extending the theoretical perspectives of reading in science and mathematics education research. International Journal of Science and Mathematics Education, 20(Suppl. 1), S1–S15. https://doi.org/10.1007/s10763-022-10322-1

    Article  Google Scholar 

  • Tsai, C.-C., Jessie Ho, H. N., Liang, J.-C., & Lin, H.-M. (2011). Scientific epistemic beliefs, conceptions of learning science and self-efficacy of learning science among high school students. Learning and Instruction, 21(6), 757–769. https://doi.org/10.1016/j.learninstruc.2011.05.002

    Article  Google Scholar 

  • Tuncay, B., Yılmaz-Tüzün, Ö., & Teksoz, G. T. (2012). Moral reasoning patterns and influential factors in the context of environmental problems. Environmental Education Research, 18(4), 485–505.

    Article  Google Scholar 

  • Wang, J.-R., & Chen, S.-F. (2016). Development and validation of an online dynamic assessment for raising students’ comprehension of science text. International Journal of Science and Mathematics Education, 14(3), 373–389. https://doi.org/10.1007/s10763-014-9575-4

    Article  Google Scholar 

  • Wang, J.-R., Chen, S.-F., Fang, I., & Chou, C.-T. (2012). Development of a test assessing elementary students’ comprehension of science texts. International Journal of Science and Mathematics Education, 10(4), 955–973.

    Article  Google Scholar 

  • Wellington, J., & Osborne, J. (2001). Language and literacy in science education. McGraw-Hill Education (UK).

  • Wilson, M. (2023). Constructing measures: An item response modeling approach. Taylor & Francis.

  • Wiyarsi, A., Çalik, M., Priyambodo, E., & Dina, D. (2023). Indonesian prospective teachers’ scientific habits of mind: A cross-grade study in the context of local and global socio-scientific issues. Science & Education, 1–27.

  • Wu, M., Adams, R., Wilson, M., & Haldane, S. (2007). ACER ConQuest 2.0: General item response modelling software [Computer program manual]. Australian Council for Educational Research.

  • Yang, F.-Y., Chang, C.-C., Chen, L.-L., & Chen, Y.-C. (2016). Exploring learners’ beliefs about science reading and scientific epistemic beliefs, and their relations with science text understanding. International Journal of Science Education, 38(10), 1591–1606.

    Article  Google Scholar 

  • Yore, L. D., Hand, B., Goldman, S. R., Hildebrand, G. M., Osborne, J. F., Treagust, D. F., & Wallace, C. S. (2004). New directions in language and science education research. Reading Research Quarterly, 39(3), 347–352.

    Google Scholar 

  • Yore, L. D., Pimm, D., & Tuan, H.-L. (2007). The literacy component of mathematical and scientific literacy. International Journal of Science and Mathematics Education, 5(4), 559–589.

    Article  Google Scholar 

  • Yore, L. D., & Tang, K.-S. (2022). Foundations, insights, and future considerations of reading in science and mathematics education. International Journal of Science and Mathematics Education, 20(Suppl. 1), S237–S260. https://doi.org/10.1007/s10763-022-10321-2

    Article  Google Scholar 

  • Zeidler, D. L. (2014). Socioscientific issues as a curriculum emphasis: Theory, research, and practice. In N. G. Lederman & S. K. Abell (Eds.), Handbook of research on science education (Vol. 2, pp. 711–740). Routledge.

  • Zeidler, D. L., & Nichols, B. H. (2009). Socioscientific issues: Theory and practice. Journal of Elementary Science Education, 21, 49–58.

    Article  Google Scholar 

  • Zhai, X. (2022). Assessing high-school students’ modeling performance on Newtonian mechanics. Journal of Research in Science Teaching, 59(8), 1313–1353.

    Article  Google Scholar 

  • Zhang, J., & Browne, W. J. (2023). Exploring Chinese high school students’ performance and perceptions of scientific argumentation by understanding it as a three-component progression of competencies. Journal of Research in Science Teaching, 60(4), 847–884.

    Article  Google Scholar 

  • Zwick, R., Thayer, D. T., & Lewis, C. (1999). An empirical Bayes approach to Mantel-Haenszel DIF analysis. Journal of Educational Measurement, 36(1), 1–28.

    Article  Google Scholar 

Download references

Funding

This project was funded by the Quality Education Fund, Hong Kong SAR Government. The name of the project is Empowering Hong Kong STEM secondary students’ reading abilities through a school-based reciprocal reading program and an online learning platform, Quality Education fund, Hong Kong SAR Government (Grant Number: 9420033).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Jack K. H. Pun.

Ethics declarations

Ethics approval

This project was approved by Human and Artefacts Ethics Sub-Committee at City University of Hong Kong (Application Number: HU-STA-00000011). Students’ and parents’ consents were sought before carrying out the study.

Conflict of interest

The authors declare no competing interests.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 255 kb)

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Cheung, K.K.C., Pun, J.K.H. & Fu, X. Development and Validation of a Reading in Science Holistic Assessment (RISHA): a Rasch Measurement Study. Int J of Sci and Math Educ (2023). https://doi.org/10.1007/s10763-023-10434-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10763-023-10434-2

Keywords

Navigation