The relationship between vegetation and the environment is a key issue in community ecology. There are theoretical and practical reasons to clarify grassland vegetation differentiation under the coupled effects of climate variables and grazing pressure, and to quantify the effects of natural and anthropogenic disturbances on vegetation processes. In this study, we quantitatively explored the effects of climate, altitude, soil, and grazing factors on Leymus chinensis community differentiation in the Xilin River Basin using a detrended canonical correspondence analysis (DCCA) and structural equation modeling (SEM). Our results can be summarized as follows: (1) The first two axes of the DCCA explained more than 67.63% of the variability; the first axis captured mainly the variance caused by grazing and thermal-hydrological conditions, while the second axis captured the variance caused by soil variables; (2) The results of the SEM indicated that sheep density had the largest total impact on vegetation differentiation, followed by climate variables; (3) We detected indirect effects of climate, altitude, and grazing on vegetation differentiation, manifested mainly by changes in soil physical and chemical properties. This work clarified the key role of grazing on L. chinensis community differentiation in the Xilin River Basin, and also revealed the importance of climatic factors in vegetation differentiation on the basin scale. Therefore, these findings will be useful to design grassland management strategies that take into account climate characteristics for the rational use and protection of grassland resources.