Item Differential in Computer Based and Paper Based Versions of a High Stakes Tertiary Entrance Test: Diagrams and the Problem of Annotation
This paper presents the results from a tertiary entrance test that was delivered to two groups of candidates, one as a paper based test and the other as a computer based test. Item level differential reveals a pattern that appears related to item type: questions based on diagrammatic stimulus show a pattern of increased difficulty when delivered on computer. Differential in performance was not present in other sections of the test and it would appear unlikely to be explained by demographic differences between the groups. It is suggested this differential is due to the inability of the candidates to freely annotate on the stimulus when delivered on computer screen. More work needs to be done on considering the role of annotation as a problem solving strategy in high-stakes testing, in particular with certain kinds of stimulus, such as diagrams.
KeywordsDiagrams computer based assessment paper based assessment high-stakes tests standardized test annotation diagrammatic assessment NAPLAN
Unable to display preview. Download preview PDF.
- 2.McGaw, B.: A test to suit the 21st century (2013), http://www.theaustralian.com.au/archive/national-affairs/a-test-to-suit-the-21st-century/story-fnd17met-1226654600162
- 5.Way, W.D., Lin, C.-H., Kong, J.: Maintaining score equivalence as tests transition online: Issues, approaches and trends. Annual meeting of the National Council on Measurement in Education, New York, NY (2008)Google Scholar
- 7.Ainsworth, S., Prain, V., Tytler, R.: Drawing to learn in science. Representations 3, 5 (2011)Google Scholar
- 8.Marshall, C.C.: Annotation: from paper books to the digital library. In: Proceedings of the Second ACM International Conference on Digital Libraries, pp. 131–140. ACM (1997)Google Scholar
- 9.Plimmer, B., Apperley, M.: Making paperless work. In: Proceedings of the 8th ACM SIGCHI New Zealand Chapter’s International Conference on Computer-Human Interaction: Design Centered HCI, pp. 1–8. ACM (2007)Google Scholar
- 10.Tversky, B.: What do sketches say about thinking. In: 2002 AAAI Spring Symposium, Sketch Understanding Workshop, Stanford University, AAAI Technical Report SS-02-08. pp. 148–151 (2002)Google Scholar
- 11.The Tests | NAP, http://www.nap.edu.au/naplan/the-tests.html
- 16.Marshall, C.C.: Reading and interactivity in the digital library: Creating an experience that transcends paper. In: Proceedings of CLIR/Kanazawa Institute of Technology Roundtable, pp. 1–20 (2005)Google Scholar