Though the feed forward neural network based on GaussNewton algorithm and its derivation will converge with order two, it is only effective toward little residual problem. In order to solve the little and large residual problems at the same time, NL2SOL algorithm is introduced and combined with the GaussNewton algorithm so as to form a feed forward neural network based on GaussNewton-NL2SOL algorithm. The application shows that this neural network can solve the residual problem properly and the convergence and stability of it performs well.