Metal parts are widely used in various fields, and their surface defects usually distribute unevenly and some characteristics are weak, which often causes missing and false detection. To solve this problem, a YOLOv5s-MD algorithm is proposed. Aiming at the problem of complex features of metal surface defects, an improved spatial pyramid pooling module is introduced to improve the deep feature extraction for small targets of different sizes. To address the problem of feature dispersion and calculation increase, a lightweight attention mechanism and the GSConv module are added to improve the model’s ability to effectively extract defect features at different sizes. For the boundary regression mismatch caused by irregular size information of metal surface defects, a loss function considering vector angle is adopted. The results show that the YOLOv5s-MD algorithm has an average accuracy of 75.3% in metal surface defect detection, which can effectively increase the detection accuracy and reduce the false detection rate for metal surface defects.