Aiming at the problems of the mainstream deep learning algorithm for crack segmentation consumes a lot of computing resources, and the traditional image processing methods have low detection accuracy and lost crack features. In order to realize the real-time detection of concrete cracks and the segmentation of cracks at the pixel level, a crack semantic segmentation model based on lightweight convolutional neural network M-Unet is proposed. Firstly, the Mobilenet_V2 lightweight network is improved, its network structure is trimmed and the activation function is optimized, and then the encoder part with huge parameters of U-Net is replaced by the improved mobilenet_V2 to realize the lightweight of the model and improve the segmentation effect of cracks. The SegCracks data set containing 5160 crack images is constructed to verify the proposed method. The experimental results show that the crack segmentation effect of the optimized M-Unet is better than the mainstream segmentation networks of U-Net, FCN8 and SegNet and the traditional image processing technology, the obtained IoU_Score is 96.10%, F1_score is 97.99%. Compared with the original U-Net, the weight file size M-Unet is reduced by 7 %, the iteration time and prediction time are reduced by 63.3% and 68.6% respectively, and the IoU_Score and F1_Score are increased by 5.79 % and 3.14 %, respectively, and the cross validation results on different open source data sets are good, which shows that the proposed network has the advantages of high accuracy, good robustness and strong generalization ability, with a view to providing a new technical means for real-time segmentation of concrete cracks.