Trajectory optimization of a quadrotor with slung payload is addressed by accounting for non-circular restriction area of obstacle and the requirement of switching between different encapsulating circles of the whole system. First， a dynamic model of the quadrotor with slung load is established based upon a unified description of the strained-relaxed tether through introducing a complementary constraint. And then， the method of R function is employed for constructing a unified description of different encapsulating circles， and the distance between the encapsulating circle and the restriction area of obstacle is calculated using a collision detection algorithm. In addition， the obstacle-avoidance trajectory optimization problem for the quadrotor with slung payload is modeled by nonlinear optimal control method. Furthermore， the Legendre-Gauss-Radau pseudospectral method is used to discretize the open-loop nonlinear optimal control problem into a nonlinear programming problem such that the optimal trajectory of the system can be obtained through numerical solutions. Finally， effectiveness of the controller is demonstrated through a numerical study.