The accuracy of modal parameters identification has the influence on the quality of dynamic characteristics analysis of mechanical structure directly. And the precision of the frequency response function estimation has the great influence on the accuracy of modal parameters identification. In the engineering the frequency response function can be estimated by the average power spectrum and cross-power spectrum with the hel Pof FFT. As the result of limited frequency resolution in the course of FFT and the noise, the accuracy of modal parameters identification is influenced by the frequency response function estimation. This paper has put forward a least-square-based optimization of frequency response function estimation in modal parameters identification. According to the optimized frequency response function, the peak value can be searched to identify the damped natural frequencies and damping ratio directly. An experiment has been made to validate the proposed method. The result indicates that the damped natural frequencies and damping ratio identified by the optimized frequency response function are closer to the true results than which without the optimized frequency response function. So the accuracy of modal parameters identification can be improved by adopting the proposed method.