Using a Sampling Strategy to Address Psychometric Challenges in Tutorial-Based Assessments

Abstract

Introduction: Tutorial-based assessment, despite providing a good match with the philosophy adopted by educational programmes that emphasize small group learning, remains one of the greatest challenges for educators working in this context. The current study was performed in an attempt to assess the psychometric characteristics of tutorial-based evaluation upon adopting a multiple sampling approach that requires minimal recording of observations. Method: After reviewing the literature, a simple 3-item evaluation form was created. The items were “Professional Behaviour,” “Contribution to Group Process,” and “Contribution to Group Content.” Explicit definition of these items was provided on an evaluation form. Twenty five tutors in five different programmes were asked to use the form to evaluate their students (N = 169) after every tutorial over the course of an academic unit. Each item was rated using a 10-point scale. Results: Cronbach’s alpha revealed an appropriate internal consistency in all five programmes. Test—retest reliability of any single rating was low, but the reliability of the average rating was at least 0.75 in all cases. The construct validity of the tool was supported by the observation of increasing ratings over the course of the academic unit and by the finding that more senior students received higher ratings than more junior students. Conclusion: Consistent with the context specificity phenomenon, the adoption of a “minimal observations often” approach to tutorial-based assessment appears to maintain better psychometric characteristics than do attempts to assess tutorial performance using more comprehensive measurement tools.