Proper modification of the involute gear could effectively improve its meshing performance, lower the noise and extend its service life. A finite element model of helical gear pair in mesh with different amounts of tip relief was built by ANSYS parameter design language (APDL). Static transmission error and time-varying meshing stiffness with the different amounts of tip relief were calculated by loaded tooth contact analysis. Then, a 12-degree of freedom dynamic model of a parallel shaft helical gear meshing was established with considering the impact of tip relief. And the time-varying meshing stiffness was introduced into the dynamic model to research the influence of tip relief on the vibration response of helical gearing. The results show that the amplitude of radial vibration and meshing force of helical gear system decrease obviously with the increase of tip relief within a certain range, but it has an increase tendency when the tip relief reaches 21 μm. The results are significant to determine the optimal gear profile modification and analyze the vibration characteristics of repaired helical gears.