The incidence and mortality of gastric cancer is steadily increasing annually around the world, which required further investigation about alternative therapy strategies. Melatonin, an indoleamine synthesized in the pineal gland, has shown dramatic anticancer effect in several cancers, however, the function of melatonin in gastric cancer needs to be characterized. In this study, we found that melatonin inhibited the growth and induced apoptosis of gastric cancer cells. microRNAs (miRNAs) have been attractive targets for many anticancer drugs. To explore the underlying molecular mechanism by which melatonin attenuated the growth of cancer cells, miRNA microarray analysis was performed to screen the miRNAs, which significantly altered after melatonin treatment. The result showed that melatonin administration enhanced the expression of miR-16-5p. Further molecular mechanism research revealed that miR-16-5p targeted Smad3 and consequently negatively regulated the abundance of Smad3. Consistently, melatonin exposure decreased the level of Smad3 and overexpression of Smad3 attenuated the inhibitory effect of melatonin in gastric cancer cells. These results uncovered the anticancer effect of melatonin and highlighted the critical roles of miR-16-5p-Smad3 pathway in melatonin-induced growth defects of gastric cancers.