The drill string in oil and gas exploration is an extremely flexible rotor system with time delay and nonsmooth effects. The drilling system is prone to lose stability, leading to severe stickslip vibration and even causing drilling failure. This research focuses on the axialtorsional coupled dynamics and timedelay stability of the drill string. Considering the internal structure damping, external damping, gravity, buoyancy, and the general top boundary, a 4degreeoffreedom drill string model is proposed. Considering the bit bounce and torsional vibration on the drill bit, multiple regenerative effects were introduced at the bitrock interface. Hence, the drilling model at hand is a nonlinear axialtorsional coupled dynamic system with a complex time delay. Then, based on the semidiscretization method, the stability criterion of the linear derived timedelay system of the system is deduced. The stability envelope is obtained on the parametric plane spanned by the rotating speed and the weight on bit. By parametric study, stability results are compared for different boundary conditions at the top of the drill string and different rock strength properties. Subsequently, nonlinear simulations are carried out in the time domain. The simulation results verify the stability analysis and reproduce the nonlinear phenomena, including the stickslip vibrations and bit bounces. Finally, the selfexcited vibrations due to the time delay is completely mitigated via an optimal tuned top boundary, which provides a simple and effective way for drilling vibration suppression. This research can be served as a basis for reducing drillstring failures in drilling engineering.