In response to the real-time scheduling problem in complex robot manufacturing cell with multiple constraints such as multiple handling robots, multiple line transfers, and buffer capacity limitations, a new two-stage real-time scheduling algorithm is proposed with the objective of minimizing the maximum completion time. The algorithm comprises an initial solution construction phase and a local search optimization phase. In the first phase, a Multi-Layer Feedback Queue (MLFQ) scheduling algorithm based on priority rules is designed by integrating the ideas of first-come-first-serve and round-robin approaches to quickly generate high-quality initial solutions. In the second phase, a simulated annealing algorithm based on reinforcement learning (Q-learning) is designed to perform local search on the initial solutions. The local search utilizes 7 neighborhood search operations and selects the best operation based on the Q-table values of the current state in the Q-learning algorithm to generate new solutions. Experimental comparisons on modified and augmented standard datasets validate the effectiveness and stability of the algorithm through improvements in optimal values, average values, and variance values.