Aiming at the low efficiency and precision of hot rail steel surface faults detecting at present,a suit of surface defect detection system of hot heavy rail based on the machine vision is put forward. Multi-CCD cameras are used to collect pictures. According to the geometric characteristics of the heavy rail and its defect characteristics of high-frequency region,six angle shot is used for heavy rail,and then various image processing technology are adopted in workstation. The system adopts improved Hough transform to get surface faults and Kohonen network to make a classification for the characteristics of low SVM training algorithm. The above key machine vision technology for detection of hot heavy rail surface defects greatly improves the speed and accuracy of testing and the detecting correct rate arrives over 85%.