Teacher-Examiners’ Explicit and Enacted Beliefs About Proficiency Indicators in National Oral Assessments

Abstract

The test of English oral proficiency is important in High-Stakes national examinations in which large numbers of teachers are involved as examiners. Although the literature shows that the reliability of oral assessments is often threatened by rater variability, to date the role of teacher beliefs in teacher-rater judgements has received little attention. This exploratory qualitative study conducted in Singapore identified teachers’ beliefs about the construct of oral proficiency for their assessment of secondary school candidates and examined the extent to which these beliefs had been enacted in real-time assessment. Seven experienced national-level examiners participated in this study. They listened to audio-recordings of four students performing an oral interview (conversation) task in a simulated examination and assessed the performance of each of them individually. Data about teachers’ thinking which revealed their underlying beliefs when assessing was elicited through Concurrent Verbal Protocol (CVP) sessions. In addition, a questionnaire was administered a month later to elicit their explicit beliefs. Findings showed that teachers possessed a range of beliefs about the construct of oral proficiency but only some of these formed the core of their expressed criteria when assessing student performance in real time. Implications for oral assessments and further research are discussed.

Appendix: Teacher Beliefs About Oral Proficiency (TeBOP)

Instruction

Use the following scale (numbers 1 through 4) to describe what you think about each of the statements below. For each statement, circle the number that gives the best description of what you believe to be true.

I consider the features listed below to be important when I assess a candidate’s oral proficiency in oral interview/conversation tasks.

Phonology
1. Stress	1	2	3	4
2. Rhythm	1	2	3	4
3. Intonation	1	2	3	4
4. Pronunciation	1	2	3	4
Language
5. Grammar	1	2	3	4
6. Vocabulary	1	2	3	4
7. Use of standard English	1	2	3	4
8. Uses a range of sentence structures correctly	1	2	3	4
Fluency
9. Hesitation	1	2	3	4
10. Repetition	1	2	3	4
11. Restructuring sentences	1	2	3	4
12. Reselecting vocabulary	1	2	3	4
Communication strategies
13. Achievement strategies to (paraphrase, circumlocution, etc.,)	1	2	3	4
14. Interaction strategies (clarification, ask for repetition, etc.)	1	2	3	4
15. Avoidance strategy (avoid unfamiliar topics)	1	2	3	4
Topical knowledge
16. Has interesting ideas	1	2	3	4
17. Elaborates ideas	1	2	3	4
18. Expresses ideas clearly	1	2	3	4
19. Gives a relevant personal response	1	2	3	4
20. Displays maturity in ideas	1	2	3	4
21. Displays breadth of knowledge	1	2	3	4
22. Displays depth of knowledge	1	2	3	4
23. Uses a range of relevant vocabulary	1	2	3	4
Discourse
24. Expresses ideas cohesively and coherently	1	2	3	4
25. Initiates discussion/conversation with the examiner	1	2	3	4
26. Concludes discussion/conversation	1	2	3	4
Personal characteristics
27. Interacts easily with the examiner	1	2	3	4
28. Enthusiastic about what he/she says	1	2	3	4
29. Responds enthusiastically to prompts	1	2	3	4
30. Shows effort	1	2	3	4
31. Good grooming	1	2	3	4
32. Confident	1	2	3	4
33. A pleasant voice	1	2	3	4

If you can, please explain the reasons for your choices in each of the categories:

Phonology

Accuracy

Fluency

Communication strategies

Topical knowledge

Discourse management

Personal characteristics

One or two other features, if any, you would like to add.

34.	1	2	3	4
35.