What causes individuals to hurt others? Since the famous case of Phineas Gage, lesions of the ventromedial prefrontal cortex (VMPFC) have been reliably linked to physically aggressive behavior. However, it is unclear whether naturally-occurring deficits in VMPFC, among normal individuals, might have widespread consequences for aggression. Using voxel based morphometry, we regressed gray matter density from the brains of 138 normal female and male adults onto their dispositional levels of physical aggression, verbal aggression, and sex, simultaneously. Physical, but not verbal, aggression was associated with reduced gray matter volume in the VMPFC and to a lesser extent, frontopolar cortex. Participants with less gray matter density in this VMPFC cluster were much more likely to engage in real-world violence. These findings suggest that even granular deficits in normal individuals' VMPFC gray matter can promote physical aggression.