Price-Haywood et al.1 compared audit-feedback vs. audit-feedback plus physician communication training [including standardized patient (SP) feedback] to improve cancer screening rates among patients with limited health literacy. Intervention physicians were rated higher in general communication about cancer risks and shared decision making related to colorectal cancer screening by the SPs who delivered the intervention. However, the intervention did not improve cancer screening rates or patient cancer screening knowledge.

Three important methodological limitations impact interpretation of the study results. First, SPs both delivered the intervention and evaluated the physicians at follow-up. Individuals delivering the intervention should not be involved in the evaluation process because of potential biases that may affect the outcome.2 This study could have instead recorded the SP visits and had the recordings reviewed by individuals blinded to study condition.

Second, although the investigators appropriately designed a group (cluster) randomized trial (GRT) to evaluate the intervention, they failed to appropriately power or analyze the data to account for this design. In a GRT, it is expected that outcomes from the same group (e.g., clinic) will be more similar than outcomes from other groups. The analyses presented do not account for the within-clinic (or physician) correlation. Ignoring positive correlation within groups leads to inaccurately small p values.3 Thus, the statistically significant difference in mammography screening rates at follow-up (favoring the control condition) may not be statistically significant.

Finally, the study was poorly designed to assess improvement in screening rates. Recruiting patients who are ineligible for some screening tests (e.g., men for breast and cervical cancer) provides no information on the impact of the intervention on screening compliance. A study of this size should have focused on one particular cancer screening and recruited only patients eligible for such screening.

The study of interventions to improve cancer screening rates especially among individuals with limited health literacy is important.4 Unfortunately, design and analytic flaws in this study may have overstated the impact of the intervention on all outcomes measured. The methodological issues raised by this study must be addressed in future studies to allow accurate conclusions about effectiveness of such interventions.