Mainstream deep learning algorithm for crack segmentation consumes a lot of computing resources while the traditional image processing methods are of low detection accuracy and lost crack features. In order to realize the real-time detection of concrete cracks and the segmentation of cracks at the pixel level, a crack semantic segmentation model based on lightweight convolutional neural network M-Unet is proposed. Firstly, the MobileNet_V2 lightweight network is improved, its network structure is trimmed and the activation function is optimized, and then the encoder part with huge parameters of U-Net is replaced by the improved MobileNet_V2 to realize the lightweight of the model and improve the segmentation effect of cracks. The SegCracks data set containing 5 160 crack images is constructed to verify the proposed method. The experimental results show that the crack segmentation effect of the optimized M-Unet is better than the mainstream segmentation networks of U-Net, FCN8 and SegNet and the traditional image processing techniques, the obtained IoU_Score is 96.10%, F1_Score is 97.99%. Compared with the original U-Net, the weight file size M-Unet is reduced by 7 %, the iteration time and prediction time are reduced by 63.3% and 68.6% respectively, and the IoU_Score and F1_Score are increased by 5.79 % and 3.14 % respectively. The cross validation results on different open source data sets are good, which shows that the proposed network has the advantages of high accuracy, good robustness and strong generalization ability.