Metal parts are widely used in various fields such as production and life, and their surface defects are not evenly distributed and some characteristics are weak, which often causes missing and false detection. To solve this problem, the YOLOv5s-MD algorithm is proposed. In this algorithm, the XSPPF module is improved, the GSConv module is integrated, and the lightweight attention mechanism is added to improve the model"s ability to effectively acquire defect features at different scales. The SIoU loss function is adopted to consider the inconsistency between the predicted frame and the anchor frame due to the irregular and large difference in the size information of metal surface defects. The detection algorithm was trained and validated in the augumented metal surface defect dataset. The results show that the average precision of defect detection mAP@0.5 reaches 75.3%. The proposed detection algorithm can effectively improve the detection precision and reduce the false detection in the detection of metal surface defects.