Impaired expression of emotion through pitch, loudness, rate, and rhythm of speech (affective prosody) is common and disabling after right hemisphere (RH) stroke. These deficits impede all social interactions. Previous studies have identified cortical areas associated with impairments of expression, recognition, or repetition of affective prosody, but have not identified critical white matter tracts. We hypothesized that: (1) differences across patients in specific acoustic features correlate with listener judgment of affective prosody and (2) these differences are associated with infarcts of specific RH gray and white matter regions. To test these hypotheses, 41 acute ischemic RH stroke patients had MRI diffusion weighted imaging and described a picture. Affective prosody of picture descriptions was rated by 21 healthy volunteers. We identified percent damage (lesion load) to each of seven regions of interest previously associated with expression of affective prosody and two control areas that have been associated with recognition but not expression of prosody. We identified acoustic features that correlated with listener ratings of prosody (hereafter "prosody acoustic measures") with Spearman correlations and linear regression. We then identified demographic variables and brain regions where lesion load independently predicted the lowest quartile of each of the "prosody acoustic measures" using logistic regression. We found that listener ratings of prosody positively correlated with four acoustic measures. Furthermore, the lowest quartile of each of these four "prosody acoustic measures" was predicted by sex, age, lesion volume, and percent damage to the seven regions of interest. Lesion load in pars opercularis, supramarginal gyrus, or associated white matter tracts (and not control regions) predicted lowest quartile of the four "prosody acoustic measures" in logistic regression. Results indicate that listener perception of reduced affective prosody after RH stroke is due to reduction in specific acoustic features caused by infarct in right pars opercularis or supramarginal gyrus, or associated white matter tracts.