Advances in Health Sciences Education

, Volume 17, Issue 3, pp 369–376 | Cite as

Faculty development on item writing substantially improves item quality

  • Naghma Naeem
  • Cees van der Vleuten
  • Eiad Abdelmohsen Alfaris


The quality of items written for in-house examinations in medical schools remains a cause of concern. Several faculty development programs are aimed at improving faculty’s item writing skills. The purpose of this study was to evaluate the effectiveness of a faculty development program in item development. An objective method was developed and used to assess improvement in faculty’s competence to develop high quality test items. This was a quasi experimental study with a pretest-midtest-posttest design. A convenience sample of 51 faculty members participated. Structured checklists were used to assess the quality of test items at each phase of the study. Group scores were analyzed using repeated measures analysis of variance. The results showed a significant increase in participants’ mean scores on Multiple Choice Questions, Short Answer Questions and Objective Structured Clinical Examination checklists from pretest to posttest (p < .0005). The effect sizes were 1.38, 3.84 and 2.20 for Multiple Choice Questions, Short Answer Questions and Objective Structured Clinical Examination, respectively. This study emphasizes that items written by faculty without faculty development are generally lacking in quality. It also provides evidence of the value of faculty development in improving the quality of items generated by faculty.


Assessment Evaluation Faculty development Quality Test items 



The authors acknowledge the contribution of Dr. Rukhsana W. Zuberi, Associate Dean Education (Aga Khan University) and Dr. Syeda Kauser Ali, Senior Lecturer (Aga Khan University) in review and pilot testing of checklists, facilitation of the course and scoring of items. Implementation of this research was supported by the Department for Educational Development, Aga Khan University, Karachi, Pakistan.


  1. Abate, M. A., Stamatakis, M. K., & Haggett, R. R. (2003). Excellence in curriculum development and assessment. American Journal of Pharmaceutical Education, 67(3) Article 89. Accessed on June 12, 2011.
  2. Case, S., & Swanson, D. (2001). Constructing written test questions for the basic and clinical sciences (pp. 19–29). Philadelphia: National Board of Medical Examiners.Google Scholar
  3. Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Hillsdale, NJ: Lawrence Earlbaum Associates.Google Scholar
  4. Darling-Hammond, L. (1997). School reform at the crossroads: Confronting the central issues of teaching. Educational Policy, 11(2), 151–166.CrossRefGoogle Scholar
  5. Darling-Hammond, L. (1999). Teacher quality and student achievement: A review of state policy evidence. Seattle, WA: University of Washington, Center of Teaching and Policy.Google Scholar
  6. Downing, S., & Haladyna, T. (1997). Test item development: validity evidence from quality assurance procedures. Applied Measurement Education, 10(1), 61–82.CrossRefGoogle Scholar
  7. Downing, S., & Haladyna, T. (2006). Handbook of test development. Hillsdale, NJ: Lawrence Erlbaum Associates.Google Scholar
  8. Fitzgerald, C. (2005). Risk management: Calculating the bottom line of developing a certification or licensure exam. From Accessed on 13 June 2010.
  9. Frary, R. B. (1995). More multiple-choice item writing do’s and don’ts. Practical Assessment, Research and Evaluation, 4(11). From Accessed June 13, 2011.
  10. Frey, B., Petersen, S., Edwards, L., et al. (2005). Item-writing rules: Collective wisdom. Teaching and Teacher Education, 21, 357–364.CrossRefGoogle Scholar
  11. Fullan, M., & Stiegelbauer, S. (1991). The new meaning of educational change (2nd ed.). New York: Teachers College Press.Google Scholar
  12. Haladyna, T. (1995). Developing and validating multiple-choice test items. Applied Psychological Measurement, 19, 205–207.CrossRefGoogle Scholar
  13. Hill, H. C. (2007). Learning in the teaching workforce. The Future of Children, 17(1), 111–128.CrossRefGoogle Scholar
  14. Jozefowicz, R. F., Koeppen, B. M., Case, S., Galbraith, R., Swanson, D., & Glew, R. (2002). The quality of in-house medical school examinations. Academic Medicine, 77(2), 156–161.CrossRefGoogle Scholar
  15. Kim, J., Chi, Y., Huensch, A., Jun, H., Li, H., & Roullion, V. (2010). A case study on an item writing process: Use of test specifications, nature of group dynamics, and individual item writers’ characteristics. Language Assessment Quarterly, 7(2), 160–174.Google Scholar
  16. Mayenga, C. (2009). Mapping item writing tasks on the item writing ability scale. From Accessed on June 13, 2010.
  17. Norman, G. R. (2003). RCT = results confounded and trivial: The perils of grand educational experiments. Medical Education, 37, 582–584.CrossRefGoogle Scholar
  18. Rodriguez, M. C. (1997). The art and science of item-writing: A meta-analysis of multiple-choice item format effects. From Accessed on June 13, 2010.
  19. Schmeiser, C. B., & Welch, C. J. (2006). Test development. In R. L. Brennan (Ed.), Educational measurement (4th ed., pp. 307–353). Washington, DC: The National Council on Measurement in Education and The American Council on Education.Google Scholar
  20. Steinart, Y., Mann, K., Centeno, A., Dolmans, D., Spencer, J., Gelula, M., et al. (2006). A systematic review of faculty development initiatives designed to improve teaching effectiveness in medical education: BEME guide no. 8. Medical Teacher, 28(6), 497–526.CrossRefGoogle Scholar
  21. Tunk, J. (2001). The effect of training on test item writing on test performance of junior high students. Educational Studies, 27, 129–142.CrossRefGoogle Scholar
  22. Wallach, P. M., Crespo, L. M., Holtzman, K. Z., Galbraith, R. M., & Swanson, D. B. (2010). Use of a committee review process to improve the quality of course. Examinations. Advances in Health Sciences Education, 11(1), 61–68.CrossRefGoogle Scholar
  23. Ward, A., & Murray-Ward, M. (1994). Guidelines for the development of item banks. Educational measurement: Issues and practice, 13, 34–39.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media B.V. 2011

Authors and Affiliations

  • Naghma Naeem
    • 1
    • 2
  • Cees van der Vleuten
    • 1
  • Eiad Abdelmohsen Alfaris
    • 1
  1. 1.King Saud University Chair for Medical Education Research and Development, Department of Family and Community Medicine (34), College of MedicineKing Saud UniversityRiyadhKingdom of Saudi Arabia
  2. 2.Department for Educational DevelopmentAga Khan UniversityKarachiPakistan

Personalised recommendations