The transmission quality and fatigue life of electro-mechanical drive systems can be significantly affected by heavy loads and broadband excitation, which readily trigger torsional vibrations. To study the dynamic behavior of a transmission system composed of both single-planet and double-planet planetary gear sets within a power coupling mechanism, a lumped-parameter dynamic model was developed. This model incorporates the high-order dynamic characteristics of irregular structures and stepped shafts. The torsional modes of individual planetary gear sets and the overall transmission system were calculated and compared with simulation results. In addition, the effects of structural parameters on the natural frequencies of the system were investigated. The results show that the average relative error in natural frequencies between the proposed dynamic model and the simulation model is 1.4%, confirming the model's accuracy. Both individual planetary gear sets and the complete transmission system exhibit three categories of vibration modes: rigid body modes, independent torsional vibration modes of the planet gears, and overall torsional vibration modes. Furthermore, the full transmission system also exhibits localized modes associated with structural irregularities. Sensitivity analysis reveals that the structural parameters most affecting the system’s natural frequency, in descending order, are the outer diameter of the input shaft, meshing stiffness, outer diameter of the output shaft, effective meshing width of the planetary gear sets, output equivalent moment of inertia, and input equivalent moment of inertia. Notably, meshing stiffness significantly influences the independent torsional modes of the planetary gears. The methodology presented in this study offers a valuable reference for analyzing torsional vibrations in multistage planetary gear transmission systems.