Shrub-encroachment onto grassland is becoming an important ecological problem in grassland ecosystems, and accurate estimation of the shrub above-ground biomass (AGB) in shrub-encroached grassland vegetation plays a significant role in research into regional ecosystem carbon cycles. Due to the dual effects of soil background noise and differences in vegetation growth structure characteristics, the traditional vegetation indices are extremely unstable for model-building involving shrub-encroached grassland AGB estimation. To solve this problem, in this study we developed a novel way by optimizing the triangular vegetation index (TVI) using Sentinel-2 remote sensing data for shrub-encroached grassland AGB estimation. The results showed that: 1) In the area dominated by herbaceous vegetation, TVI calculated using a combination of green, red-edge and near-infrared (, and ) performed best with an R2 of 0.684; in the area dominated by shrub vegetation, the TVI again performed best with R2= 0.368. 2) When analyzing the sensitivity of the 12 commonly used vegetation indexes to soil noise, the enhanced vegetation index (EVI) was the most sensitive to soil noise in the area dominated by herbaceous vegetation; in the area dominated by shrub vegetation, the modified soil adjusted vegetation index (MSAVI) was the most sensitive. 3) In the area dominated by herbaceous vegetation, the optimized vegetation index grassland triangular vegetation index (GTVI) performed better than TVI with the value of (coefficient of determination cross validation) increased by 0.153 and the value of decreased by 12.222 g·m-2; in the area dominated by shrub vegetation, GTVI performed better than TVI and the value increased 0.029, while the (root mean square error cross validation) decreased 1.684 g·m-2. 4) The estimation results acquired by GTVI showed the highest accuracy when compared with the results estimated by the commonly used 12 vegetation indices. The results of this study are expected to provide a scientific basis and reference AGB estimation in shrub-encroached grassland using vegetation indices extracted from remote sensing data.