Investigating Program Differences in ACGME General Surgery Milestone 2.0 Ratings: Medical Knowledge and Practice-Based Learning and Improvement

Objective: Previous studies indicate that ACGME milestone ratings may reflect substantial program level variation. The objective of this study is to investigate the program level variation in general surgery medical knowledge (MK) and practice-based learning and improvement (PBLI) milestone ratings. Design: This study is a retrospective observational study of milestone ratings. Descriptive statistics were calculated at both the individual and program level. Intraclass correlation coefficients (ICCs) were analyzed for each PGY level and MK and PBLI subcompetency. Within program uniform rating practices were characterized using descriptive statistics and program size between uniform and nonuniform rating programs was compared with the Wilcoxon Rank Sum test. Participants: The sample included PGY1-PGY5 2020-2021 evaluation data for 7,581 residents across 328 general surgery programs. Results: There was a wide range in average program milestone rating at each PGY level. ICC results indicated that program level differences accounted for 32.3% to 58.2% of the total variance across PGY levels for MK1, 37.4% to 61.8% for MK2, 36.3% to 64.8% for PBLI1, and 35.1% to 63.8% for PBLI2. Program variations most heavily influenced PGY1 and PGY5 ratings. Up to 36% of programs practiced uniform rating, and uniform scores most frequently corresponded to PGY level. Programs practicing uniform rating were significantly smaller than those with variation in resident ratings (p < 0.001). Discussion: Our study found that 1 to two-thirds of the variance in milestone ratings for MK and PBLI can be attributed to program variations. These variations may be attributed to program differences in CCC rating practices. A high percentage of programs rate residents uniformly corresponding to their PGY level. Further research is needed to investigate the consistency and fairness of clinical competency committee ratings practices, mitigate uniform rating, and improve the degree to which milestone ratings reflect individual resident competency as opposed to the residency program.