Testing for Multilevel Dimensionality: A Higher-Order Factor Analysis of a Script Concordance Test

Abstract

Background

To date, few script concordance test (SCT) research studies have empirically investigated whether SCTs measure a single test dimension of clinical reasoning. Prior analyses have been inconclusive and unsuccessful at identifying exam dimensions through simple first-order factor analyses. Therefore, the principle objective of this work was to explore the higher-order factor structure of a problem-solving SCT to determine whether the dimensionality of the test conformed to a multilevel construct arrangement.

Methods

This retrospective data analysis utilized scores from medical students (n = 522) who took a SCT in their fourth year of undergraduate training. Higher-order factor analyses were conducted on six different SCT scoring methods. In addition, Schmid-Leiman solutions were conducted to evaluate the proportion of variance unique to a given level of the model.

Results

Five of the six scoring methods yielded a factor structure with three first-order factors and one second-order factor. The total variance in the first-order factors explained by the second-order factor was notable at greater than 40 %. However, Schmid-Leiman solutions unveiled the deceptiveness of the higher-order models in that, across the various scoring methods, very few items contributed unique variance to the second- or first-order factors. Ideally, all items would have contributed unique variance to one or more levels of the model.

Conclusions

In this study, SCT scores did not conform to a clear interpretable higher-order factor structure suggesting that SCTs may not measure the meaningful clinical reasoning constructs they are thought to measure. An explanation for these findings is provided, and recommendations for revising the SCT format and scoring procedures are proposed.

Sample 1: A Sample Vignette and Questions from a typical Emergency Medicine SCT

A 22-year-old female presents to the Emergency Department complaining of lower abdominal pain for the last 12 h. She describes the pain as sharp in the right lower quadrant. She has some nausea with one episode of vomiting. Her last menstrual period was 6 weeks prior, but she is irregular. She has only one sexual partner, who is male.

Given the above case scenario, answer the following questions:

Diagnostic questions

	If you were thinking of the following diagnosis…	…and you find the following evidence….	…the hypothesis becomes…
1	Appendicitis	Normal WBC count	−2 −1 0 +1 +2
2	Tubo-ovarian abscess	Unilateral right adenexal tenderness with palpable mass on pelvic exam	−2 −1 0 +1 +2
3	Urinary tract infection	History of dysuria and frequency	−2 −1 0 +1 +2

1. −2 ,highly unlikely; −1, less likely than before; 0, neither more nor less likely; +1, more likely than before; +2, very likely

Investigational questions

	If you were considering asking for…	…and you find the following evidence….	…this investigation becomes…
4	Pelvic ultrasound	Serum hCG = 650 mIU/ml	−2 −1 0 +1 +2

1. −2, not useful at all; −1, less useful; 0, neither more nor less useful; +1, useful; +2, absolutely necessary
2. Therapeutic questions

	If you were considering asking for…	…and you find the following evidence….	…this treatment becomes…
5	IV morphine	Positive urine pregnancy test	−2 −1 0 +1 +2
6	Ceftriaxone and azithromycin	Bilateral adnexal and cervical motion tenderness	−2 −1 0 +1 +2

1. −2, contraindicated totally or almost totally; −1, not useful, possibly detrimental; 0, neither more nor less useful; +1, useful; +2, absolutely necessary