For a ISA HEV,the transmission gear shaft dynamic model is established and the 3-D modal analysis is carfled out. Based on above work, the structure of the gear shaft is analysed, then the diameter of the gear shaft is taken as a design variable. With first order optimization,low inherent frequencies are optimized in order to avoid resonance and decrease NVH.