The transmission quality and fatigue life of electro-mechanical drive system will be reduced due to heave load and broad-spectrum excitation, which readily trigger torsional vibration. In order to study the dynamic characteristics of transmission system composed of single-planet and double-planet planetary gear sets in power coupling mechanism, high-order dynamic characteristics of irregular structures and stepped shafts were considered, the transmission system’s lumped-parameter dynamic model was established, the torsional modal of individual planetary gear sets and transmission system were calculated and compared with simulation results, the effects of structure parameters on the natural frequency of the transmission system were investigated. The results shows that the average relative error of natural frequency between dynamic model and simulation model is 1.4%. Both free planetary gear sets and transmission system contain three types of vibration modes which are rigid body mode, independent torsional vibration modes of planet gears and overall torsional vibration modes, the transmission system also includes local modes of irregular structures. The descending order of structure parameters according to their influence on natural frequency are outside diameter of input shaft, meshing stiffness, outside diameter of output shaft, effective meshing width of planetary gear sets, output equivalent moment of inertia, input equivalent moment of inertia. The meshing stiffness has significant influence on the independent torsional vibration modes of planet gears. The method discussed here can provide a reference for torsional vibration analysis of multistage planetary gear transmission system.