In recent years, deep neural network and computer vision techniques have played an increasingly important role in structural health monitoring. In this paper, deep learning technology is used to identify the PE sheath damage through the image data of bridge stay cable damage collected by UAV. This paper aims to realize the intelligent and rapid identification of local damage on the surface of stay cables of long-span bridges on devices with low computational ability, and to solve the problems of relatively low computational efficiency and large scale of model parameters in the current research of traditional deep convolution neural network. A lightweight region proposal convolution neural network model is proposed. Firstly, the theoretical basis of region proposal network and its lightweight improvement method is introduced, and the necessity of lightweight model processing is analyzed. It reduces the performance requirements of devices for model training and prediction under the premise of ensuring identification accuracy, achieving the purpose of saving computational resources and time. Secondly, the problem of insufficient data of damage samples is solved by multiple means of data augmentation. The contrast experiment and the analysis of statistical results verify the superiority of the lightweight neural network model. The results show that the lightweight network can improve the complexity and quantity of calculation of the model to a large extent under the premise of a small sacrifice of recognition accuracy. It effectively expands the practicability of the neural network in engineering applications.