The stop-band characteristic of dielectric layer PBG structure in waveguide filter was analyzed using precise integration in symplectic system,and the filter was optimized on response surface methodology. The transverse section was discretized by edge element. The export stiff matrices of a dielectric layer and an air layer can be calculated by the precision integration based on Riccati equations in Hamilton system. The export stiff matrices of a period can be obtained by the combination of substructures, and the whole structure can be solved. Optimization model of the filter was established on the basis of analysis. The approximate expression was found between the design parameters and the objective function by response surface methodology. The model was solved using quadratic programming and the optimal parameters were obtained. Example demonstrated that it was effective.