The existence of toothside backlash in gear transmission system induces rich nonlinear dynamical behaviors. Considering the nonlinear dynamic model of a two-degrees-of-freedom gear transmission system, the bifurcation and chaos of the system were analyzed by using the simple cell mapping method. Firstly, the global characteristics of the nonlinear gear system were analyzed, and the attractor and attracting domain were obtained. The results show that, with changing excitation frequency, the system exhibits coexistence of multiperiodic solutions and coexistence of periodic and chaotic motions. Furthermore, the phase diagram of the trajectory of the system was compared with that of the Poincaré section. It is shown that, under different initial conditions, the system presents different periodic or chaotic motions. Therefore, based on the numerical results by the simple cell mapping method, ideal system responses, such as periodic motions, can be realized by reasonably selecting the initial conditions, when certain parameters make the system undergoes complicated responses.