A finite element vibration analysis model of speed-increase gearbox was established, and modal analysis of gear system has been performed by adopting Lanczos method. Considering the time-varying mesh stiffness, tooth error and impact exaction, we used the direct integration method to obtain the vibration response of surface nodes of gearbox. Taking twelve main structure parameters on the housing as the design variables, the volume of the gearbox as the state variable, RMS of vibration acceleration on the surface of the gearbox as the objective function, we used the zero order and the first order optimization method to acquire the optimal solution of design variables. The results indicate that resonance phenomenon doesn't occur and the gearbox satisfies the static condition both before and after optimization. The objective function has reduced by 37.5%, and significant reduce of vibration response of the computing points of the gearbox can also be found after optimization, and the maximum reduction is 54%.