Surface corrosion constitutes a prevalent defect form in steel structures, which can undermine the section of components and deteriorate their mechanical properties. Thus, it is imperative to monitor and assess the corrosion damage of steel structures. A lightweight semantic segmentation model for rust images, based on HL-DeepLabV3+, has been proposed to address the challenges of reading errors, texture misjudgments, and color discrepancies caused by visual fatigue during manual inspections. This model aims to improve segmentation accuracy while reducing the parameter count associated with traditional deep learning networks used in rust recognition. Firstly, the coding portion of the backbone network was replaced with a lightweight MobileNetV3 network to minimize the number of model parameters. Secondly, the strip pool module and pyramid pool module were incorporated into the void space pyramid pool module, enabling the network to capture the long-range dependence of isolated rust regions and eliminate the interference in the background region. Finally, the attention mechanism was introduced to make the network focus more on the pixel region that plays a decisive role in rust classification and enhance the feature expression ability of rust. The test results indicate that: Compared with the original model, the size of the HL-DeepLabV3+ model is reduced by 66.73%, and the Accuracy, mIoU, MPA, and F1 scores are increased by 2.54%, 10.11%, 6.79%, and 6.15% respectively. Moreover, the effect is superior to the classical UNet, FCN, SegNet, and DeepLabV3+ semantic segmentation models. This not only realizes the lightweight of the model but also enhances the segmentation accuracy of rust damage.