A vision system based on the service robot is involved. In the system, picture signal is acquired by the picture sensor OV7635. Frame memory AL422B is used as data buffer memory, while CPLD controlled the time order DSP performed. In the software system of image processing, to accomplish color image segmentation and recognition, the threshold vector judgment and improved seed-fill algorithm is introduced, and the image geometric moment is calculated during the segmenting. In order to achieve a vision servo system which composed of image-based feedback and adaptive compensation, the deduced matrix-based Jacobian from the image moment is taken as image feature. Adopted the TFT LCD is adopted to straightly disolav the result of vision recognition and vision tracing.