Through coupling vibration analysis of the transmission system of a 1.5 MW wind turbine gearbox, a torsional vibration model of a gear drive system in wind turbines is established. The fourthorder RungeKutta integration method is applied to calculate the dynamic response of a gear system caused by wind loads, timevarying meshing stiffness, and system damping. The harmonic balance method is used to find the analytic solution. Based on the solution, a mathematical model for optimum gear system design is established. The model aims at minimizing the vibration acceleration of the planet gear and the total mass of the gear transmission system. The optimization toolbox in MATLAB program is adopted to obtain the optimal solution. A calculated example shows that the primary natural frequency of the gear system has been raised, the dynamic properties of the gear train have been improved remarkably, and the total mass of the gear train has been decreased.