Cognitive decline is a significant challenge for middle-aged and elderly individuals. The association of exercise with cognitive decline is inconsistent and may be influenced by many physiological factors. This study aimed to explore the mediating factors in the relationship between physical exercise and cognitive decline. This prospective cohort study included 3,153 middle-aged and older adults with normal baseline cognitive function (2011) in the China Health and Retirement Longitudinal Study Cohort. Cognitive decline was assessed after four years (2015). Exercise level was assessed by exercise intensity, frequency, and metabolic equivalent. The results showed that any level of exercise intensity or frequency was associated with decreased risk of cognitive decline, but the association was stronger in subjects ≥ 60 years or males. Vigorous exercise might positively correlate with cognitive decline through reducing body mass index, and this association was mainly present in females. In conclusion, active exercise is positively associated with cognitive function in middle-aged and elderly adults. Exercise may affect cognitive function through body mass index, but the specific mechanisms require further investigation.