The low-pressure rotor system of Geared Turbofan (GTF) is more complicated due to the star-gear transmission structure, which makes it possess different dynamic characteristics with the traditional engine. Therefore, the vibration problem caused by the gear flexible rotor coupling system is more complicated. Based on the structural characteristics of the low-pressure rotor, the finite element model of the threepoint rotor system with star-gear transmission structure is established. By applying the eccentric mass to the highspeed rotor disk, the coupled vibration characteristics of the rotor system under unbalanced vector excitation are studied. Conducted relevant test analysis. The results show that under the double-plate unbalanced excitation, due to the coupling effect between the imbalances, the vibration of each part of the rotor system is weakened. And compared with the center gear, the meshing force between the ring gear and the star gear is larger, and the influence of the unbalanced initial phase is more significant. In addition, when the phase difference is 180°, the imbalance is more harmful to the rotor system.