Skip to main content
SpringerLink
Log in

Script concordance testing in continuing professional development: local or international reference panels?

  • Published:
Advances in Health Sciences Education Aims and scope Submit manuscript

Abstract

Context The PRACTICUM Institute has developed large-scale international programs of on-line continuing professional development (CPD) based on self-testing and feedback using the Practicum Script Concordance Test© (PSCT). Aims To examine the psychometric consequences of pooling the responses of panelists from different countries (composite panels) and the effect of increasing composite panel size. Method 97 cardiologists in Mexico answered a set of 62 PSCT cases/305 questions. A local panel was recruited in Mexico (n = 7). Other panelists were recruited in Argentina (n = 10) and Brazil (n = 11). Together they constituted a composite panel of 28 experts. Random panels of reference of increasing sizes (5, 10, 15, 20, and 25) were generated. Participants’ scores were computed for each panel sample. Units of analysis were means of participants’ scores per case. Discrimination, ranking and reliability of the scores obtained with each panel were estimated. Descriptive statistics, Pearson correlation coefficient, generalizability analysis, computation of Cronbach’s alpha were used in the analyses. Results Correlation coefficients between the local and the composite panels ranged from 0.951 to 0.981. Cronbach alpha coefficient values were above 0.85 for all panels. The value of the relative G coefficient from the generalizability analyses varied from 0.91 to 0.93, indicating very high and stable ranking of participants, though absolute value of scores increased with increasing composite panel size. Conclusions In CPD environments, and with panel members selected with the highest standards, composite panels can be used. Panels of all sizes yielded high psychometric qualities. Absolute scores should be interpreted with caution.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  • Bursztejn, A. C., Cuny, J. F., Adam, J. L., Sido, L., Schmutz, J. L., de Korwin, J. D., et al. (2011). Usefulness of the script concordance test in dermatology. Journal of the European Academy of Dermatology and Venereology, 25, 1471–1475.

    Article  Google Scholar 

  • Cardinet, J., Johnson, S., & Pini, G. (2012). Applying generalisability theory using EduG. New York, London: Routledge, Taylor and Francis Group.

    Google Scholar 

  • Charlin, B., & van der Vleuten, C. (2004). Standardized assessment of reasoning in contexts of uncertainty: the script concordance approach. Evaluation in the Health Professions, 27, 304–319.

    Article  Google Scholar 

  • Charlin, B., Brailovsky, C., Leduc, C., & Blouin, D. (1998). The Diagnosis Script Questionnaire: A new tool to assess a specific dimension of clinical competence. Advances in Health Sciences Education, 3, 51–58.

    Article  Google Scholar 

  • Charlin, B., Boshuizen, H. P., Custers, E. J., & Feltovich, P. J. (2007). Scripts and clinical reasoning. Medical Education, 41, 1178–1184.

    Article  Google Scholar 

  • Dory, V., Gagnon, R., Vanpee, D., & Charlin, B. (2012). How to construct and implement script concordance tests: insights from a systematic review. Medical Education, 46, 552–563.

    Article  Google Scholar 

  • Elstein, A. S. (1993). Beyond multiple-choice questions and essays: The need for a new way to assess clinical competence. Academic Medicine, 68, 244–249.

    Article  Google Scholar 

  • Fournier, J. P., Demeester, A., & Charlin, B. (2008). Script concordance tests: Guidelines for construction. BMC Medical Informatic Decision Making, 8, 18, http://www.biomedcentral.com/1472-6947/8/18.

  • Gagnon, R., Charlin, B., Coletti, M., Sauve, E., & van der Vleuten, C. (2005). Assessment in the context of uncertainty: How many members are needed on the panel of reference of a script concordance test?Medical. Education, 39, 284–291.

    Google Scholar 

  • Gantelet, M. (2008). Impact du panel de référence sur les résultats d’un Test de Concordance de Script (TCS) développé en formation initiale des sages-femmes. Dissertation for the completion of a diploma of the Ecole des cadres sages-femmes. Dijon: Ecole de Cadres Sages-Femmes.

  • Hornos, E., Pleguezuelos, E. M., Brailovsky, C. A., Harillo, L. D., Dory, V., & Charlin, B. (2012). The Practicum Script Concordance Test: an online continuing professional development format to foster reflection on clinical practice. Journal of Continuing Education in the Health Professions, in press.

  • Norcini, J. J., & Shea, J. A. (1990). The effect of level of expertise on answer key development. Academic Medecine, 65(Suppl), S15–S16.

    Article  Google Scholar 

  • Norman, G. R. (1985). Objective measurement of clinical performance. Medical Education, 19, 43–47.

    Article  Google Scholar 

  • Sibert, L., Charlin, B., Corcos, J., Gagnon, R., Grise, P., & van der Vleuten, C. (2002). Stability of clinical reasoning assessment results with the Script Concordance test across two different linguistic, cultural and learning environments. Medical Teacher, 24, 522–527.

    Article  Google Scholar 

Download references

Acknowledgments

The authors would like to thank Leandro Harillo for his help in collecting the data and preparing this manuscript.

Author information

Authors and Affiliations

  1. PRACTICUM Institute of Applied Research in Health Sciences Education, Madrid, Spain

    E. M. Pleguezuelos, E. Hornos, P. Malagrino & C. A. Brailovsky

  2. Fonds de la Recherche Scientifique, FNRS and Institute of Health and Society, Université catholique de Louvain, Brussels, Belgium

    V. Dory

  3. CPASS (Centre de Pédagogie Appliquée aux Sciences de la Santé), Faculty of Medicine, University of Montreal, CP 6128 Succursale centre-ville, Montreal, QC, H3C 3J7, Canada

    R. Gagnon & B. Charlin

  4. College of Family Physicians of Canada, Montreal, Canada

    C. A. Brailovsky

Authors
  1. E. M. Pleguezuelos
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. E. Hornos
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. V. Dory
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. R. Gagnon
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. P. Malagrino
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. C. A. Brailovsky
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. B. Charlin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to B. Charlin.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Pleguezuelos, E.M., Hornos, E., Dory, V. et al. Script concordance testing in continuing professional development: local or international reference panels?. Adv in Health Sci Educ 18, 779–785 (2013). https://doi.org/10.1007/s10459-012-9427-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10459-012-9427-3

Keywords

Search

Navigation