In order to reveal the vibration mechanism of the engine doghub transmission system, the working principle of the engine doghub system was analyzed. Using the energy method and the Adams software, the dynamic models of the engine doghub transmission system were established. The dynamic modeling and analysis methods were proposed for three-jaw curved contact system of the doghub. By comparing the axial force response results of the two models under rated working condition, the correctness of the dynamic models was verified. To reduce the vibration and impact of the doghub transmission system, the influences of different torques and axial preload on the axial impact force and displacement of the doghub system were studied. Results show that the dynamic response characteristics of the system under different operating conditions are significantly different. As the torque increases from 350 N·m to 500 N·m, the axial impact force of the system increases, and the Axial displacement nearly doubled. As the axial preload gradually increases from 6 000 N to 8 000 N, the peak value of the system axial impact force increases, and the mean value changes slightly after stabilization, and the axial displacement decreases accordingly. Under the condition of rated torque, an appropriate small axial preload is adopted to reduce the axial impact force on the doghub transmission system and relieve the vibration and impact of the system.