Abstract
The quality of items written for in-house examinations in medical schools remains a cause of concern. Several faculty development programs are aimed at improving faculty’s item writing skills. The purpose of this study was to evaluate the effectiveness of a faculty development program in item development. An objective method was developed and used to assess improvement in faculty’s competence to develop high quality test items. This was a quasi experimental study with a pretest-midtest-posttest design. A convenience sample of 51 faculty members participated. Structured checklists were used to assess the quality of test items at each phase of the study. Group scores were analyzed using repeated measures analysis of variance. The results showed a significant increase in participants’ mean scores on Multiple Choice Questions, Short Answer Questions and Objective Structured Clinical Examination checklists from pretest to posttest (p < .0005). The effect sizes were 1.38, 3.84 and 2.20 for Multiple Choice Questions, Short Answer Questions and Objective Structured Clinical Examination, respectively. This study emphasizes that items written by faculty without faculty development are generally lacking in quality. It also provides evidence of the value of faculty development in improving the quality of items generated by faculty.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Abate, M. A., Stamatakis, M. K., & Haggett, R. R. (2003). Excellence in curriculum development and assessment. American Journal of Pharmaceutical Education, 67(3) Article 89. http://www.xula.edu/cop/documents/Assessment-Curriculum/Excellence%20in%20Curriculum%20Development%20and%20Assessment.pdf. Accessed on June 12, 2011.
Case, S., & Swanson, D. (2001). Constructing written test questions for the basic and clinical sciences (pp. 19–29). Philadelphia: National Board of Medical Examiners.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Hillsdale, NJ: Lawrence Earlbaum Associates.
Darling-Hammond, L. (1997). School reform at the crossroads: Confronting the central issues of teaching. Educational Policy, 11(2), 151–166.
Darling-Hammond, L. (1999). Teacher quality and student achievement: A review of state policy evidence. Seattle, WA: University of Washington, Center of Teaching and Policy.
Downing, S., & Haladyna, T. (1997). Test item development: validity evidence from quality assurance procedures. Applied Measurement Education, 10(1), 61–82.
Downing, S., & Haladyna, T. (2006). Handbook of test development. Hillsdale, NJ: Lawrence Erlbaum Associates.
Fitzgerald, C. (2005). Risk management: Calculating the bottom line of developing a certification or licensure exam. From http://www.caveon.com/articles/fitzgerald3.htm. Accessed on 13 June 2010.
Frary, R. B. (1995). More multiple-choice item writing do’s and don’ts. Practical Assessment, Research and Evaluation, 4(11). From http://pareonline.net/getvn.asp?v=4&n=11. Accessed June 13, 2011.
Frey, B., Petersen, S., Edwards, L., et al. (2005). Item-writing rules: Collective wisdom. Teaching and Teacher Education, 21, 357–364.
Fullan, M., & Stiegelbauer, S. (1991). The new meaning of educational change (2nd ed.). New York: Teachers College Press.
Haladyna, T. (1995). Developing and validating multiple-choice test items. Applied Psychological Measurement, 19, 205–207.
Hill, H. C. (2007). Learning in the teaching workforce. The Future of Children, 17(1), 111–128.
Jozefowicz, R. F., Koeppen, B. M., Case, S., Galbraith, R., Swanson, D., & Glew, R. (2002). The quality of in-house medical school examinations. Academic Medicine, 77(2), 156–161.
Kim, J., Chi, Y., Huensch, A., Jun, H., Li, H., & Roullion, V. (2010). A case study on an item writing process: Use of test specifications, nature of group dynamics, and individual item writers’ characteristics. Language Assessment Quarterly, 7(2), 160–174.
Mayenga, C. (2009). Mapping item writing tasks on the item writing ability scale. From http://www.ocs.sfu.ca/fedcan/index.php/csse2009/csse2009/paper/viewFile/1966/625. Accessed on June 13, 2010.
Norman, G. R. (2003). RCT = results confounded and trivial: The perils of grand educational experiments. Medical Education, 37, 582–584.
Rodriguez, M. C. (1997). The art and science of item-writing: A meta-analysis of multiple-choice item format effects. From http://www.edmeasurement.net/aera/papers/artandscience.pdf. Accessed on June 13, 2010.
Schmeiser, C. B., & Welch, C. J. (2006). Test development. In R. L. Brennan (Ed.), Educational measurement (4th ed., pp. 307–353). Washington, DC: The National Council on Measurement in Education and The American Council on Education.
Steinart, Y., Mann, K., Centeno, A., Dolmans, D., Spencer, J., Gelula, M., et al. (2006). A systematic review of faculty development initiatives designed to improve teaching effectiveness in medical education: BEME guide no. 8. Medical Teacher, 28(6), 497–526.
Tunk, J. (2001). The effect of training on test item writing on test performance of junior high students. Educational Studies, 27, 129–142.
Wallach, P. M., Crespo, L. M., Holtzman, K. Z., Galbraith, R. M., & Swanson, D. B. (2010). Use of a committee review process to improve the quality of course. Examinations. Advances in Health Sciences Education, 11(1), 61–68.
Ward, A., & Murray-Ward, M. (1994). Guidelines for the development of item banks. Educational measurement: Issues and practice, 13, 34–39.
Acknowledgments
The authors acknowledge the contribution of Dr. Rukhsana W. Zuberi, Associate Dean Education (Aga Khan University) and Dr. Syeda Kauser Ali, Senior Lecturer (Aga Khan University) in review and pilot testing of checklists, facilitation of the course and scoring of items. Implementation of this research was supported by the Department for Educational Development, Aga Khan University, Karachi, Pakistan.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Naeem, N., van der Vleuten, C. & Alfaris, E.A. Faculty development on item writing substantially improves item quality. Adv in Health Sci Educ 17, 369–376 (2012). https://doi.org/10.1007/s10459-011-9315-2
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10459-011-9315-2