A dynamic model of transmission system for a marine gear is established considering time-varying mesh stiffness, mesh damping and tooth composite error. The solution efficiency is improved by taking contact line length change of gear pair instead of instantaneous gear mesh stiffness change and making mesh damping and tooth surface equivalent to viscous damping. Multi-objective dynamic optimization design is put into practice to improve the marine gear transmission system using vibration acceleration and mass as target function. The results show that the optimization method can effectively reduce the vibration level and mass of marine gear.