Faculty development on item writing substantially improves item quality
- 806 Downloads
The quality of items written for in-house examinations in medical schools remains a cause of concern. Several faculty development programs are aimed at improving faculty’s item writing skills. The purpose of this study was to evaluate the effectiveness of a faculty development program in item development. An objective method was developed and used to assess improvement in faculty’s competence to develop high quality test items. This was a quasi experimental study with a pretest-midtest-posttest design. A convenience sample of 51 faculty members participated. Structured checklists were used to assess the quality of test items at each phase of the study. Group scores were analyzed using repeated measures analysis of variance. The results showed a significant increase in participants’ mean scores on Multiple Choice Questions, Short Answer Questions and Objective Structured Clinical Examination checklists from pretest to posttest (p < .0005). The effect sizes were 1.38, 3.84 and 2.20 for Multiple Choice Questions, Short Answer Questions and Objective Structured Clinical Examination, respectively. This study emphasizes that items written by faculty without faculty development are generally lacking in quality. It also provides evidence of the value of faculty development in improving the quality of items generated by faculty.
KeywordsAssessment Evaluation Faculty development Quality Test items
The authors acknowledge the contribution of Dr. Rukhsana W. Zuberi, Associate Dean Education (Aga Khan University) and Dr. Syeda Kauser Ali, Senior Lecturer (Aga Khan University) in review and pilot testing of checklists, facilitation of the course and scoring of items. Implementation of this research was supported by the Department for Educational Development, Aga Khan University, Karachi, Pakistan.
- Abate, M. A., Stamatakis, M. K., & Haggett, R. R. (2003). Excellence in curriculum development and assessment. American Journal of Pharmaceutical Education, 67(3) Article 89. http://www.xula.edu/cop/documents/Assessment-Curriculum/Excellence%20in%20Curriculum%20Development%20and%20Assessment.pdf. Accessed on June 12, 2011.
- Case, S., & Swanson, D. (2001). Constructing written test questions for the basic and clinical sciences (pp. 19–29). Philadelphia: National Board of Medical Examiners.Google Scholar
- Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Hillsdale, NJ: Lawrence Earlbaum Associates.Google Scholar
- Darling-Hammond, L. (1999). Teacher quality and student achievement: A review of state policy evidence. Seattle, WA: University of Washington, Center of Teaching and Policy.Google Scholar
- Downing, S., & Haladyna, T. (2006). Handbook of test development. Hillsdale, NJ: Lawrence Erlbaum Associates.Google Scholar
- Fitzgerald, C. (2005). Risk management: Calculating the bottom line of developing a certification or licensure exam. From http://www.caveon.com/articles/fitzgerald3.htm. Accessed on 13 June 2010.
- Frary, R. B. (1995). More multiple-choice item writing do’s and don’ts. Practical Assessment, Research and Evaluation, 4(11). From http://pareonline.net/getvn.asp?v=4&n=11. Accessed June 13, 2011.
- Fullan, M., & Stiegelbauer, S. (1991). The new meaning of educational change (2nd ed.). New York: Teachers College Press.Google Scholar
- Kim, J., Chi, Y., Huensch, A., Jun, H., Li, H., & Roullion, V. (2010). A case study on an item writing process: Use of test specifications, nature of group dynamics, and individual item writers’ characteristics. Language Assessment Quarterly, 7(2), 160–174.Google Scholar
- Mayenga, C. (2009). Mapping item writing tasks on the item writing ability scale. From http://www.ocs.sfu.ca/fedcan/index.php/csse2009/csse2009/paper/viewFile/1966/625. Accessed on June 13, 2010.
- Rodriguez, M. C. (1997). The art and science of item-writing: A meta-analysis of multiple-choice item format effects. From http://www.edmeasurement.net/aera/papers/artandscience.pdf. Accessed on June 13, 2010.
- Schmeiser, C. B., & Welch, C. J. (2006). Test development. In R. L. Brennan (Ed.), Educational measurement (4th ed., pp. 307–353). Washington, DC: The National Council on Measurement in Education and The American Council on Education.Google Scholar