The X chromosome shows a special interaction between demographic factors and genetic variation, and the analysis of X-linked genomic variation can therefore provide insights into the unique effects of demography and selection on the horse genome that cannot be readily detected by autosomal markers. Debao (DB) ponies have experienced intense selective pressure for the development of their small stature (<106 cm at adult height). To identify selective sweeps on the X chromosome of the DB pony, we performed a genome-wide scan of three Chinese horse breeds using an Equine SNP70 BeadChip. Using Yili and Mongolian horses (>134 cm at adult height) as reference groups, both FST and XP-EHH revealed that five regions on the X chromosome were under strong selection, resulting in 95 overlapping genes. Seven of these genes, SMS, PHEX, ACSL4, CHRDL1, CACNA1F, DKC1 and CDKL5, are involved in bone development, growth hormone secretion and fat deposition. The region showing the strongest selection pressure was located at the position of 86.6-87.5 Mb. The subsequent genome-wide association analysis of the adult height of three Chinese horse breeds detected the two most significant SNPs in the same region, and these two SNPs overlapped with the gene CHRDL1. As a member of the bone morphogenetic protein (BMP) superfamily, CHRDL1 antagonizes the function of BMP4 and plays an important role in embryonic bone formation and cartilage generation. Our results provide new insights into the X-linked selection in Chinese Debao pony.