Growth deviating from the norm during childhood has been associated with anorexia nervosa (AN) and obesity later in life. In this study, we examined whether polygenic scores (PGSs) for AN and BMI are associated with growth trajectories spanning the first two decades of life. AN PGSs and BMI PGSs were calculated for participants of the Avon Longitudinal Study of Parents and Children (ALSPAC; n = 8,654). Using generalized (mixed) linear models, we associated PGSs with trajectories of weight, height, body mass index (BMI), fat mass index (FMI), lean mass index (LMI), and bone mineral density (BMD). Female participants with AN PGSs one standard deviation (SD) higher had, on average, 0.004% slower growth in BMI between the ages 6.5 and 24 years and a 0.4% slower gain in BMD between the ages 10 and 24 years. Higher BMI PGSs were associated with faster growth for BMI, FMI, LMI, BMD, and weight trajectories in both sexes throughout childhood. Female participants with both a high AN PGS and a low BMI PGS showed slower growth compared to those with both a low AN PGS and a low BMI PGS. We conclude that AN PGSs and BMI PGSs have detectable sex-specific effects on growth trajectories. Female participants with a high AN PGS and low BMI PGS likely constitute a high-risk group for AN, as their growth was slower compared to their peers with high PGSs on both traits. Further research is needed to better understand how the AN PGS and the BMI PGS co-influence growth during childhood and whether a high BMI PGS can mitigate the effects of a high AN PGS.