To fully understand and accurately predict the motion characteristics of two mass dynamics systems, this paper constructs a solution procedure of residue harmonic balance for the study of two?degree?of?freedom motion systems, taking two?mass dynamic system connected with fixed boundaries as an example. This solution procedure combines the advantages of harmonic balance and homotopy method, and the higher?order approximations only depend on the initial harmonic approximation, and they don′t need to be adjusted according to the previous approximations. The results show that the frequencies based on the presented second?order residue harmonic balance are more accurate than the results obtained by other existing methods, and the relative errors are reduced in different degrees. Moreover, the presented steady responses are in good agreement with the numerical ones. It is shown that this method can be applied widely to other nonlinear oscillation problems.