Predictive Validity Evidence for Medical Education Research Study Quality Instrument Scores: Quality of Submissions to JGIM’s Medical Education Special Issue
Deficiencies in medical education research quality are widely acknowledged. Content, internal structure, and criterion validity evidence support the use of the Medical Education Research Study Quality Instrument (MERSQI) to measure education research quality, but predictive validity evidence has not been explored.
To describe the quality of manuscripts submitted to the 2008 Journal of General Internal Medicine (JGIM) medical education issue and determine whether MERSQI scores predict editorial decisions.
Design and Participants
Cross-sectional study of original, quantitative research studies submitted for publication.
Study quality measured by MERSQI scores (possible range 5–18).
Of 131 submitted manuscripts, 100 met inclusion criteria. The mean (SD) total MERSQI score was 9.6 (2.6), range 5–15.5. Most studies used single-group cross-sectional (54%) or pre-post designs (32%), were conducted at one institution (78%), and reported satisfaction or opinion outcomes (56%). Few (36%) reported validity evidence for evaluation instruments. A one-point increase in MERSQI score was associated with editorial decisions to send manuscripts for peer review versus reject without review (OR 1.31, 95%CI 1.07–1.61, p = 0.009) and to invite revisions after review versus reject after review (OR 1.29, 95%CI 1.05–1.58, p = 0.02). MERSQI scores predicted final acceptance versus rejection (OR 1.32; 95% CI 1.10–1.58, p = 0.003). The mean total MERSQI score of accepted manuscripts was significantly higher than rejected manuscripts (10.7 [2.5] versus 9.0 [2.4], p = 0.003).
MERSQI scores predicted editorial decisions and identified areas of methodological strengths and weaknesses in submitted manuscripts. Researchers, reviewers, and editors might use this instrument as a measure of methodological quality.
- Dauphinee WD, Wood-Dauphinee S. The need for evidence in medical education: the development of best evidence medical education as an opportunity to inform, guide, and sustain medical education research. Acad Med. 2004;79(10):925–30. CrossRef
- Wartman SA. Research in medical education: the challenge for the next decade. Acad Med. 1994;69(8):608–14. CrossRef
- Shea JA, Arnold L, Mann KV. A RIME perspective on the quality and relevance of current and future medical education research. Acad Med. 2004;79(10):931–8. CrossRef
- Lurie SJ. Raising the passing grade for studies of medical education. JAMA. 2003;290(9):1210–2. CrossRef
- Carline JD. Funding medical education research: opportunities and issues. Acad Med. 2004;79(10):918–24. CrossRef
- Regehr G. Trends in medical education research. Acad Med. 2004;79(10):939–47. CrossRef
- Cook DA, Beckman TJ. Current concepts in validity and reliability for psychometric instruments: theory and application. Am J Med. 2006;119(2):166.e7–16. CrossRef
- Chen FM, Bauchner H, Burstin H. A call for outcomes research in medical education. Acad Med. 2004;79(10):955–60. CrossRef
- Whitcomb ME. Using clinical outcomes data to reform medical education. Acad Med. 2005;80(2):117. CrossRef
- Reed DA, Cook DA, Beckman TJ, Levine RB, Kern DE, Wright SM. Association between funding and quality of published medical education research. JAMA. 2007;298(9):1002–9. CrossRef
- Prystowsky JB, Bordage G. An outcomes research perspective on medical education: the predominance of trainee assessment and satisfaction. Med Educ. 2001;35(4):331–3. CrossRef
- Beckman TJ, Cook DA, Mandrekar JN. What is the validity evidence for assessments of clinical teaching? J Gen Intern Med. 2005;20(12):1159–64. CrossRef
- Downing SM. Validity: on meaningful interpretation of assessment data. Med Educ. 2003;37(9):830–7. CrossRef
- American Educational Research Association, American Psychological Association, National Council on Measurement in Education. Standards for educational and psychological testing. Washington, DC: American Educational Research Association; 1999:9–24.
- Cullen DJ, Macaulay A. Consistency between peer reviewers for a clinical specialty journal. Acad Med. 1992;67(12):856–59. CrossRef
- Schroter S, Tite L, Hutchings A, Black N. Differences in review quality and recommendations for publication between peer reviewers suggested by authors or by editors. JAMA. 2006;295(3):314–17. CrossRef
- Prideaux D. Researching the outcomes of educational interventions: a matter of design. BMJ. 2002;324:126–7. CrossRef
- Cook DA, Bordage G, Schmidt HG. Description, justification and clarification: a framework for classifying the purposes of research in medical education. Med Educ. 2008;42(2):128–33. CrossRef
- Colliver JA. The research enterprise in medical education. Teach Learn Med. 2003;15(3):154–5. CrossRef
- Cook DA, Beckman TJ, Bordage G. Quality of reporting of experimental studies in medical education: a systematic review. Med Educ. 2007;41(8):737–45. CrossRef
- Stacy R, Spencer J. Assessing the evidence in qualitative medical education research. Med Educ. 2000;34(7):498–500. CrossRef
- Cote L, Turgeon J. Appraising qualitative research articles in medicine and medical education. Med Teach. 2005;27(1):71–5. CrossRef
- Predictive Validity Evidence for Medical Education Research Study Quality Instrument Scores: Quality of Submissions to JGIM’s Medical Education Special Issue
- Open Access
- Available under Open Access This content is freely available online to anyone, anywhere at any time.
Journal of General Internal Medicine
Volume 23, Issue 7 , pp 903-907
- Cover Date
- Print ISSN
- Online ISSN
- Additional Links
- medical education research
- research quality
- research methods
- Industry Sectors
- Author Affiliations
- 1. Division of Primary Care Internal Medicine, Mayo Clinic College of Medicine, Rochester, MN, USA
- 2. Division of General Internal Medicine, Mayo Clinic College of Medicine, Rochester, MN, USA
- 3. Department of Medicine, Johns Hopkins University School of Medicine, Baltimore, MD, USA