The differential equation which governs the behavior of the gear transmission system of 1.5 MW windturbine is established. The external excitation caused by wind speed fluctuation is discussed and the internal excitation due to timevarying mesh stiffness and comprehensive error is also analyzed. The calculating formulations of the harmonic balance method of nonlinear dynamic equations are presented. Then, the multiobjective dynamic optimization model is developed to minimize the value of vibration acceleration and the overall volume. The optimization of a practical example using mixeddiscrete combined programming is performed. The result shows that the proposed method of modeling and optimization design can effectively reduce the wind turbines gearbox vibration levels and weight.