To overcome the detection difficulties of the surface defects when milling a hot heavy rail,a suite of detection system for on-line surface defect is developed. To select optical environment according to radiation of heavy rail and characteristics of light,use multi-linear array CCD cameras to parallel multi-angle collect images of whole surface of hot heavy rail,and obtain images through an image adaptive preprocessing to meet requirements of defect detection. For the condition that the defects extracting causing by continuity of the surface defects of hot heavy rail is hard to realize through traditional image segmentation, an examining algorithm of image relevance between pixel lines is proposed, using pixel de-differentiation and statistical variance operations to extract defect. This algorithm is demonstrated to be better than traditional edge detection,and the system has achieved good results in the practical application of a rail beam plant.