Brain alterations are hypothesized to be present in patients with chronic whiplash-associated disorders (CWAD). The aim of this case-control study was to examine alterations in cortical thickness and white matter (WM) structure, and the presence of brain microhemorrhages in a patient group encountering chronic neck pain of traumatic origin (i.e., CWAD) when compared with a patient group characterized by nontraumatic chronic neck pain [i.e., chronic idiopathic neck pain (CINP)], and healthy controls. Furthermore, we aimed to investigate associations between brain structure on one hand and cognitive performance and central sensitization (CS) on the other hand. T1-weighted, diffusion-weighted and T2*-weighted magnetic resonance images of the brain were acquired in 105 women (31 controls, 37 CINP, 37 CWAD) to investigate regional cortical thickness, WM structure, and microhemorrhages, respectively. Next, cognitive performance, and CS encompassing distant hyperalgesia and conditioned pain modulation (CPM) efficacy were examined. Cortical thinning in the left precuneus was revealed in CWAD compared with CINP patients. Also, decreased fractional anisotropy, together with increased values of mean diffusivity and radial diffusivity could be observed in the left cingulum hippocampus and tapetum in CWAD compared with CINP, and in the left tapetum in CWAD patients compared with controls. Moreover, the extent of WM structural deficits in the left tapetum coincided with decreased CPM efficacy in the CWAD group. This yields evidence for associations between decreased endogenous pain inhibition, and the degree of regional WM deficits in CWAD. Our results emphasize the role of structural brain alterations in women with CWAD compared with CINP.