In order to reveal the vibration mechanism of the engine doghub transmission system, based on the analysis of the working principle of the engine doghub system, the dynamic models of the engine doghub transmission system were established with the energy method and the Adams software. The dynamic modeling and analysis methods were proposed for three-jaw curved contact system of the doghub. By comparing the axial force response results of the two models under rated working condition, the correctness of the dynamic models was verified. To reduce the vibration and impact of the doghub transmission system, the influences of different torques and axial preloads on the axial impact force and displacement of the doghub system were studied. Results show that the dynamic response characteristics of the system under different operating conditions were significantly different. As the torque increased from 350 N·m to 500 N·m, the axial impact force of the system increased, and the axial displacement nearly doubled. As the axial preload gradually increased from 6 000 N to 8 000 N, the peak value of the system axial impact force increased, and the mean value changed slightly after stabilization, and the axial displacement decreased accordingly, suggesting that under the condition of rated torque, an appropriate small axial preload is beneficial to reducing the axial impact force on the doghub transmission system and relieving the vibration and impact of the system.