Skip to main content
SpringerLink
Log in

Development and Evaluation of the Diagnostic Power for a Computer-Based Two-Tier Assessment

  • Published:
Journal of Science Education and Technology Aims and scope Submit manuscript

Abstract

This study adopted a quasi-experimental design with follow-up interview to develop a computer-based two-tier assessment (CBA) regarding the science topic of electric circuits and to evaluate the diagnostic power of the assessment. Three assessment formats (i.e., paper-and-pencil, static computer-based, and dynamic computer-based tests) using two-tier items were conducted on Grade 4 (n = 90) and Grade 5 (n = 86) students, respectively. One-way ANCOVA was conducted to investigate whether the different assessment formats affected these students’ posttest scores on both the phenomenon and reason tiers, and confidence rating for an answer was assessed to diagnose the nature of students’ responses (i.e., scientific answer, guessing, alternative conceptions, or knowledge deficiency). Follow-up interview was adopted to explore whether and how the various CBA representations influenced both graders’ responses. Results showed that the CBA, in particular the dynamic representation format, allowed students who lacked prior knowledge (Grade 4) to easily understand the question stems. The various CBA representations also potentially encouraged students who already had learning experience (Grade 5) to enhance the metacognitive judgment of their responses. Therefore, CBA could reduce students’ use of test-taking strategies and provide better diagnostic power for a two-tier instrument than the traditional paper-based version.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

References

  • Adadan E, Savasci F (2012) An analysis of 16–17-year-old students’ understanding of solution chemistry concepts using a two-tier diagnostic instrument. Int J Sci Educ 34(4):513–544

    Article  Google Scholar 

  • Arslan HO, Cigdemoglu C, Moseley C (2012) A three-tier diagnostic test to assess pre-service teachers’ misconceptions about global warming, greenhouse effect, ozone layer depletion, and acid rain. Int J Sci Educ 34(11):1667–1686

    Article  Google Scholar 

  • Ausubel DP (1968) Educational psychology: a cognitive view. Holt, Rinehart & Winston, New York, NY

    Google Scholar 

  • Brunvand S (2010) Best practices for producing video content for teacher education. Contemp Issues Technol Teach Educ 10(2):247–256

    Google Scholar 

  • Caleon I, Subramaniam R (2010) Development and application of a three-tier diagnostic test to assess students’ understanding of waves. Int J Sci Educ 32(7):939–961

    Article  Google Scholar 

  • Carlton K (1999) Teaching electric current and electrical potential. Phys Educ 34(6):341–345

    Article  Google Scholar 

  • Chiu MH, Lin JW (2005) Promoting fourth graders’ conceptual change of their understanding of electric current via multiple analogies. J Res Sci Teach 42(4):429–464

    Article  Google Scholar 

  • Chiu JL, Linn MC (2014) Supporting knowledge integration in chemistry with a visualization-enhanced inquiry unit. J Sci Educ Technol 23:37–58

    Article  Google Scholar 

  • Chiu MH, Guo CJ, Treagust DF (2007) Assessing students’ conceptual understanding in science: an introduction about a national project in Taiwan. Int J Sci Educ 29(4):379–390

    Article  Google Scholar 

  • Chu HE, Treagust DF, Chandrasegaran AL (2009) A stratified study of students’ understanding of basic optics concepts in different contexts using two-tier multiple-choice items. Res Sci Technol Educ 27(3):253–265

    Article  Google Scholar 

  • Duit R, Treagust DF (2012) How can conceptual change contribute to theory and practice in science. In: Fraser BJ, Tobin K, McRobbie C (eds) Second international handbook of science education. Springer, Dordrecht, The Netherlands, pp 107–118

    Chapter  Google Scholar 

  • Eshach H (2003) Small-group interview-based discussions about diffused shadows. J Sci Educ Technol 12(3):261–275

    Article  Google Scholar 

  • Griffard PB, Wandersee JH (2001) The two-tier instrument on photosynthesis: what does it diagnose? Int J Sci Educ 23(10):1039–1052

    Article  Google Scholar 

  • Hasan S, Bagayoko D, Kelley EL (1999) Misconceptions and the certainty of response index (CRI). Phys Educ 34(5):294–299

    Article  Google Scholar 

  • Hennessy S, Wishart J, Whitelock D, Deaney R, Brawn R, La Velle L, McFarlane A, Ruthven K, Winterbottom M (2007) Pedagogical approaches for technology-integrated science teaching. Comput Educ 48(1):137–152

    Article  Google Scholar 

  • Kanli U (2014) A study on identifying the misconceptions of pre-service and in-service teachers about basic astronomy concepts. Eurasia J Math Sci Technol Educ 10(5):471–479

    Article  Google Scholar 

  • Levy D (2013) How dynamic visualization technology can support molecular reasoning. J Sci Educ Technol 22:702–717

    Article  Google Scholar 

  • Lin JW (2008) A comparison study between the coherence of across-grade students’ mental models in electricity and curriculum sequence. J Educ Psychol 31(3):53–79

    Google Scholar 

  • Lin JW (2015) Do skilled elementary teachers hold scientific conceptions and can they accurately predict the type and source of students’ preconceptions of electric circuits? Int J Sci Math Educ. doi:10.1007/s10763-015-9635-4

    Google Scholar 

  • Lin JW, Chiu MH (2009) An across-grade study to investigate the evolutionary processes of students’ cognitive characters in series connection. J Res Educ Sci 54(4):139–170

    Google Scholar 

  • Lin JW, Wu YL (2011) Investigating the impact of dynamic representations with interactive response system on diagnosing students’ alternative conceptions of simple and series connections. J Educ Psychol 34(1):79–107

    Google Scholar 

  • Lin JW, Wu YL (2013) Application of a diagnostic instrument with item confidence to explore across graders’ understanding of simple and series circuits and their sources. J Res Educ Sci 58(2):25–56

    Google Scholar 

  • Lin JW, Wu YL, Lin YC (2011) The impact of integrating dynamic representations in a two-tier test on diagnosing students’ answering in simple and series circuits. Res Dev Sci Educ Q 61:25–50

    Google Scholar 

  • Loh ASL, Subramaniam R, Tan KCD (2014) Exploring students’ understanding of electrochemical cells using an enhanced two-tier diagnostic instrument. Res Sci Technol Educa 32(3):229–250

    Article  Google Scholar 

  • Lundeberg M, Mohan L (2009) Context matters: Gender and cross-culture differences in confidence. In: Hacker DJ, Dunlosky J, Graesser AC (eds) Handbook of metacognition in education. Routledge, New York, NY, pp 222–239

    Google Scholar 

  • Ministry of Education (2012) Grade 1-9 curriculum guidelines. Ministry of Education, Taipei, Taiwan

    Google Scholar 

  • Mislevy RJ, Behrens JT, Bennett RE, Demark SF, Frezzo DC, Levy R, Robinson DH, Rutstein DW, Shute VJ, Stanley K, Winters FI (2010). On the roles of external knowledge representations in assessment design. J Technol Learn Assess 8(2):1–58. Retrieved from http://www.jtla.org

  • Organization for Economic Co-operation and Development (2010) PISA computer-based assessment of student skills in science. Organization for Economic Co-operation and Development, Paris, France

    Google Scholar 

  • Peşman H, Eryilmaz A (2010) Development of a three-tier test to assess misconceptions about simple electric circuits. J Educ Res 103(3):208–222

    Article  Google Scholar 

  • Sanger MJ, Greenbowe TJ (2000) Addressing standing misconceptions concerning electron flow in aqueous solutions with instruction including computer animations and conceptual change strategies. Int J Sci Educ 22(5):521–537

    Article  Google Scholar 

  • Schwartz RN, Milne C, Homer BD, Plass JL (2013) Designing and implementing effective animations and simulations for chemistry learning. In: Suits JP, Sanger MJ (eds) Pedagogic roles of animations and simulations in chemistry courses. American Chemical Society, Washington, DC, pp 43–76

    Chapter  Google Scholar 

  • Skopeliti I, Vosniadou S (2007) Reasoning with external representations in elementary astronomy. In: Vosniadou S, Kayser D, Protopapas A (eds) Proceedings of the 2nd European cognitive science conference. Delphi, Greece, pp 244–249

    Google Scholar 

  • Treagust DF (1988) Development and use of diagnostic tests to evaluate students’ misconceptions in science. Int J Sci Educ 10(2):159–169

    Article  Google Scholar 

  • Tsai CH, Chen HY, Chou CY, Lain KD (2007) Current as the key concept of Taiwanese students’ understandings of electric circuits. Int J Sci Educ 29(4):483–496

    Article  Google Scholar 

  • Tsui CY, Treagust DF (2010) Evaluating secondary students’ scientific reasoning in genetics using a two-tier diagnostic instrument. Int J Sci Educ 32(8):1073–1098

    Article  Google Scholar 

  • Wang TH, Chiu MH, Lin JW, Chou CC (2013) Diagnosing students’ mental models via the web-based mental models diagnosis (WMMD) system. Br J Educ Technol 44(2):E45–E48

    Article  Google Scholar 

  • Wu HC, Yeh TK, Chang CY (2010) The design of an animation-based test system in the area of Earth sciences. Br J Educa Technol 41(3):E53–E57

    Article  Google Scholar 

  • Wu HK, Kuo CY, Jen TH, Hsu YS (2015) What makes an item more difficult? Effects of modality and type of visual information in a computer-based assessment of scientific inquiry abilities. Comput Educ 85:35–48

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Curriculum Design and Human Potentials Development, National Dong-Hwa University, No. 1, Sec. 2, Da Hsueh Rd., Shoufeng, 97401, Hualien, Taiwan, ROC

    Jing-Wen Lin

Authors
  1. Jing-Wen Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jing-Wen Lin.

Appendices

Appendix 1

See Fig. 3.

Fig. 3
figure 3

Two-tier diagnostic assessment: paper-and-pencil test

Full size image

Appendix 2

See Fig. 4.

Fig. 4
figure 4

Two-tier diagnostic assessment: static representation format

Full size image

Appendix 3

See Fig. 5.

Fig. 5
figure 5

Two-tier diagnostic assessment: Dynamic representation format

Full size image

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lin, JW. Development and Evaluation of the Diagnostic Power for a Computer-Based Two-Tier Assessment. J Sci Educ Technol 25, 497–511 (2016). https://doi.org/10.1007/s10956-016-9609-5

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10956-016-9609-5

Keywords

Search

Navigation