Polymorphism is pervasive in molecular solids. While computational predictions of the molecular polymorphic landscape have improved significantly, identifying which polymorphs are preferentially accessed and experimentally stable remains a challenge. We report a framework that correlates short intermolecular contacts with polymorphic stability. The presence of short contacts between neighboring molecules prevents structural rearrangement and stabilizes the packing arrangement, even when the stabilized polymorph is not enthalpically favored. In the absence of such intermolecular short contacts, the molecules have added degrees of freedom for structural rearrangement, and solid-solid polymorphic transformations occur readily. Starting with a series of core-halogenated naphthalene tetracarboxylic diimides, we establish this framework with the packing polymorphs of more than 20 compounds, ranging from molecular semiconductors to pharmaceutics and biological building blocks. This framework, widely applicable across molecular solids, can help refine computational predictions by identifying the polymorphs that are kinetically stable.