Pepper disease detection is of great significance for taking effective prevention and control measures in time and reducing the risk of pepper production due to disease. Currently, pepper disease detection technology based on deep learning is still lacking. Based on this, this paper proposed an improved Faster R-CNN pepper disease detection method. The image set of pepper disease was firstly established, then the data of the image set was augmented, and the sample data set of pepper disease was labeled. Then the Faster R-CNN network was pre-trained, and the weight parameters were fine-tuned on the pepper disease data set through the transfer learning mechanism, its mAP reached 89.74%. It can be seen that the detection of pepper diseases by Faster R-CNN has problems of missing detection and low efficiency. In this paper, the network is improved by MobileNetv2 optimization network model combination and regional candidate box optimization, the experimental results of the improved Faster R-CNN network and the classical Faster R-CNN network are compared and analyzed, the results show that: The improved Faster R-CNN detection efficiency is significantly improved, detection speed can reach 10FPS and mAP up to 90.01%, and the missed diseases could also be identified and detected by the network, confirming the feasibility of the improved Faster R-CNN network model for pepper disease detection.