Non-dominated sorting genetic algorithm II (NSGA-II) has the shortcomings of insufficient diversity, prematurity and local convergence in solving the multi-objective optimal scheduling problem in flexible job shop. In this study, an improved NSGA-II algorithm based on reinforcement learning (RLNSGA-II) is proposed. To avoid NSGA-II to fall into the problem of local convergence, a two-population evolution strategy is introduced. The sex determination method is used to split the population into two populations, and different cross mut-ation operators are used in the evolution process to increase the local and global search capabilities of the algorithm. In order to solve the problem of insufficient diversity caused by the NSGA-II elite strategy, multiple diversity metrics are integrated, and reinforcement learning is used to dynamically optimize the split ratio parameters in the population iteration process to maintain diversity and improve algorithm convergence performance. Finally, simulation experiments and performance analysis are carried out through Kacem standard calculation examples, verifying the effectiveness and superiority of RLNSGA-II.