Aiming at the problems of low efficiency, unsatisfactory detection effect and poor safety of manual detection methods for building exterior wall cracks, a crack detection method based on aerial images and computer vision was proposed. Firstly, the Unmanned Aerial Vehicle (UAV) was used to collect the crack images through aerial photography around the buildings, and a crack dataset was constructed. Secondly, the U-Net was optimized to solve the problems of discontinuous segmentation of slender cracks as well as the missed and false detection under complex backgrounds. The encoder was replaced with pre-trained ResNet50 to improve the feature expression ability of the model. An improved Atrous Spatial Pyramid Pooling (ASPP) module was added to obtain multi-scale context information. The improved loss function was used to deal with the problem of extremely uneven distribution of positive and negative samples in crack images. Experiments show that the improved U-Net model solved the problems existing in the original model; the IoU and F1_score were increased by 3.53% and 4.18%, respectively. Compared with the classical segmentation model, the improved model has the best crack segmentation performance. Compared with manual detection methods, it can efficiently, accurately, and safely detect building exterior wall cracks.