We aimed to assess whether and how changes in brain volume and increases in white matter hyperintensity (WMH) volume over three years predict gait speed and its change independently of demographics, vascular risk factors and physical status. We analyzed 443 individuals from the Lothian Birth Cohort 1936, at mean age 73 and 76 years. Gait speed at age 76 was predicted by age, grip strength and body mass index at mean age 73, three-year brain volume decrease and WMH volume increase, explaining 26.1% of variance. Decline in gait speed to age 76 was predicted by the same five variables explaining 40.9% of variance. In both analyses, grip strength and body mass index explained the most variance. A clinically significant decline in gait speed (≥ 0.1 m/s per year) occurred in 24.4%. These individuals had more structural brain changes. Brain volume and WMH changes were independent predictors of gait dysfunction and its three-year change, but the impact of malleable physical factors such as grip strength or body mass index was greater.