Dear Editor-in-chief

Despite the increasing popularity of primary care sports medicine fellowships, as evidenced by the more than two-fold increase in family medicine sports medicine fellowships from a total of 31 accredited programs during the 1998/1999 academic year (ACGME, 1998) to 63 during the 2003/2004 academic year (ACGME, 2006), there are few empirical studies to support the efficacy of such programs. To the best of our knowledge, no studies have been conducted to assess the impact of primary care sports medicine fellowships on family medicine residents' learning of non-musculoskeletal sports medicine topics. Rigorous evaluations of the outcomes of such programs are helpful to document the value of such programs to both the lay public and interested medical residents. In order to evaluate such programs, it is helpful to apply the same objective standards to residents trained across multiple programs. Hence, we would like to know if there is a learning effect with respect to non-musculoskeletal sports medicine topics identified on yearly administered American Board of Family Medicine (ABFM) in-training exams (ITE) to family medicine residents in family medicine residency programs in the United States with and without primary care sports medicine fellowship programs.

Review and approval for the research proposal was granted by the ABFM, who also allowed access to the required data. Permission to study and report only non-musculoskeletal sports medicine topics excluding musculoskeletal topics was granted at the time due to other ongoing projects at the ABFM involving musculoskeletal topics. ABFM allowed us access to examinations from 1998 to 2003. We were given copies of each exam and records of responses to each item (correct or incorrect) by each examinee (examinees were anonymous) for each year.

For each year, each examinee was classified by the ABFM as either (a) belonging to a program that contained a sports medicine fellowship, or (b) not belonging to a program that contained a sports medicine fellowship. In order to protect anonymity, we did not receive other identifying information about candidates, such as demographics or whether participants belonged to a specific or common program. Thus, we could not group examinees by such variables as race, sex, or specific residency program.

Faculty and graduates of the Halifax Sports Medicine Fellowship program at the Halifax Medical Center in Daytona Beach, Florida were asked to sort each examination question into (a) non-musculoskeletal sports medicine questions and (b) general family medicine questions on the ABFM ITE. Examples of non-musculoskeletal sports medicine questions included topics such as concussion, female triad, altitude medicine, cardiovascular conditions, etc. All other questions (except musculoskeletal medicine items) were categorized as general family medicine questions. A total of seven faculty and graduates of the sports medicine fellowship completed the sorting task. All evaluators held board certification by the ABFM with Certificate of Added Qualifications (CAQ) in Sports Medicine at the time of evaluation. Only identified questions with unanimous agreement by all 7 evaluators were used for data analysis. Table 1 shows the number of agreed-upon questions of each type for each year. As can be seen from Table 1, data from five different examinations were available to examine the impact of the fellowship on exam performance. For each examinee, we computed two total correct scores, one for the non-musculoskeletal sports medicine items, and one for general family medicine items. The specific items change each year (1998 to 2003), so that each year had to be considered separately. Although each of the five examinations allowed for the assessment and creation of scales for both non-musculoskeletal sports medicine and general family medicine knowledge, the number and nature of questions differed across years. Different people were examined across years as well. Therefore, descriptive statistics such as the means, standard deviations, and reliabilities of the scales were not equal across years. Therefore, we analyzed data separately by year, and then combined the results across years using meta-analysis. We first discuss the logic of analyzing the data for a single year, and then present the logic of combining the analyses.

We expected that the residents in programs with sports medicine fellowships would show superior performance on the non-musculoskeletal sports medicine items. However, because assignment to fellowship was not random, we wanted to control for any possible differences in general family medicine knowledge that might exist between those residents who did and did not have a sports medicine fellowship at their residency program. Therefore, we treated scores on the family medicine scale as a covariate. We computed analysis of covariance (fellowship being a categorical independent variable) with non-musculoskeletal sports medicine items as the dependent variable. The results allow for a statistical test of the effect of sports medicine fellowship while holding general family medicine knowledge constant. In other words, we applied a statistical control for self-selection into groups. We present results both with and without statistical control (i.e., both with and without the covariate) because statistical control in the absence of random assignment to treatment, results in a very conservative test of the treatment effect when the treatment and covariate are correlated.

To combine the studies, we used the method recommended by Hedges and colleagues (Hedges and Olkin, 1985; Hedges and Vevea, 1998). For each year, we first transformed the raw data to standardized scores by subtracting the variable's mean and dividing by the variable's standard deviation, so that all transformed variables had a mean of zero and a standard deviation of one. We then computed the analysis of covariance for each year and found the standardized regression weight for fellowship along with its standard error. The inverse of the square of the standard error for each study served as the weighting factor to find a weighted average across years. For the global significance test of the fellowship effect, we compared the weighted average against its standard error (this is the analysis with statistical control). We also computed sample size weighted average correlations among the study variables (this is the analysis without statistical control).

Study results are shown by year in Table 1. The table shows (by year) the number of items in each of the two scales, Cronbach's alpha reliability estimates for each scale, the number of examinees in the sports medicine and control groups, and the correlation between the non-musculoskeletal scale and group membership, which was coded so that a positive correlation means that the sports medicine group had higher scores than the control group. The average correlations across years for all study variables are shown in Table 2. As can be seen in Table 2, there is a small but significant correlation between fellowship participation and both family medicine scores and non-musculoskeletal sports medicine scores. The result of the meta-analysis was a weighted mean effect (regression coefficient) of 0.025 (p < 0. 05), a value slightly smaller than the average correlation between fellowship and the non- musculoskeletal sports medicine scale shown in Table 2. Thus, the statistical adjustment for differences in general family medicine scores had very little effect. Meta-analysis of the fellowship regression coefficient indicated that the results were somewhat heterogeneous (Q with 4 df = 34.56, p < 0.05; the random-effects variance component was 0.0007), so a random-effects model was assumed and used to compute the overall mean effect (of 0.025).

Hunter and Schmidt (Hunter and Schmidt, 2004) provided a method of meta-analysis that allows for the correction of observed effect sizes for reliability of measurement. When the data in Table 1 were subjected to their method, the weighted average correlation corrected for reliability in the measure of non-musculoskeletal sports medicine items was 0.07, which is still small, but noticeably larger than either the weighted regression coefficient (0.025) or the weighted average correlation (0.031). This Hunter and Schmidt estimate is not adjusted (statistically controlled) for differences in scores on the general family medicine scale. We did not make the adjustment for this analysis because techniques for meta-analysis are not well adapted to regression analysis with adjustments for reliability of measurement.