ABSTRACT: Mammals have evolved different tooth phenotypes that are hypothesized to be associated with feeding habits. However, the genetic basis for the linkage has not been well explored. In this study, we investigated 13 tooth-related genes, including seven enamel-related genes (AMELX, AMBN, ENAM, AMTN, ODAM, KLK4 and MMP20) and six dentin-related genes (DSPP, COL1A1, DMP1, IBSP, MEPE and SPP1), from 63 mammals to determine their evolutionary history. Our results showed that different evolutionary histories have evolved among divergent feeding habits in mammals. There was stronger positive selection for eight genes (ENAM, AMTN, ODAM, KLK4, DSPP, DMP1, COL1A1, MEPE) in herbivore lineages. In addition, AMELX, AMBN, ENAM, AMTN, MMP20 and COL1A1 underwent accelerated evolution in herbivores. While relatively strong positive selection was detected in IBSP, SPP1, and DSPP, accelerated evolution was only detected for MEPE and SPP1 genes among the carnivorous lineages. We found positive selection on AMBN and ENAM genes for omnivorous primates in the catarrhini clade. Interestingly, a significantly positive association between the evolutionary rate of ENAM, ODAM, KLK4, MMP20 and the average enamel thickness was found in primates. Additionally, we found molecular convergence in some amino acid sites of tooth-related genes among the lineages whose feeding habit are similar. The positive selection of related genes might promote the formation and bio-mineralization of tooth enamel and dentin, which would make the tooth structure stronger. Our results revealed that mammalian tooth-related genes have experienced variable evolutionary histories, which provide some new insights into the molecular basis of dietary adaptation in mammals.
