A rotor dynamic analysis method for rotor aeroelasticity with three-dimensional blade tip was developed, and applied to analyze its aeroelastic characteristics. Based on the medium-size deformation beam theory and the free wake model, the joint transfer matrix employed in the blade kinematic and the deformation compatibility principle was utilized. The influence of the sweep angle and anhedral angle of the blade tip on the rotor aeroelastic characteristics was analyzed in detail. The results showed that the forward/aft swept configurations at the blade tips can result in the coupling effect in the flap and torsion directions, while the dihedral/anhedral configurations at the blade tips can lead to the coupling effect in the lag and torsion directions. Especially under high-speed conditions, the elastic torsion of the blades is significantly affected by the changes in the three-dimensional blade tip configurations.