Quantum genetic algorithm (QGA) has advantages of small population size with good algorithm performance, fast convergent rate and powerful ability of global search. In order to acquire high power density and low cost in-wheel motor of electric vehicle,based on the quantum genetic algorithm, a designed outer-rotor permanent magnet synchronous in-wheel motor model with 8 designed variables and 5 constraints was built to optimize the effective quality, material cost and power consumption. The results show that the effective quality, material cost and power consumption of the motor are decreased and the efficiency of the motor is improved. The results of finite element analysis are close to those calculated by quantum genetic algorithm,which can satisfy the using requirements of driving in-wheel motor electric vehicle. Therefore, the QGA is an effective and feasible algorithm in optimization design of in-wheel motor.