The outlier and the measurement that an outlier does not fit the theoretical model in the regression problems are defined. The relationship between the theoretical model and the regression model in the regression problem is analyzed. An approximate theorem is proposed and verified by deleting outlier one by one to construct SVR to approximate the theoretical model. An algorithm of detecting outliers in the SVR problems is constructed based on the approximate theorem. The theoretical analysis of the convergence and effectiveness of the proposed algorithm is given. Then, the step-by-step search algorithm is introduced to improve the outlier removing algorithm to remove outliers in SVR with large-scale samples. The theoretical analysis shows that the improved algorithm is convergent and effective. Finally, the samples produced by two test functions and the samples in UCI data set are used for simulation, and the results show that the proposed algorithm is effective and robust.