To improve the living environment is a major livelihood issue, and energy saving and emission reduction is a major national strategy. Therefore, it is necessary to seek a reasonable design method which could not only meet the needs of residents in the indoor thermal comfort and reduce the building energy consumption of residential building. Based on the genetic algorithm, a multi-objective optimization model of residential building design is established which can optimize the design to increase the indoor thermal comfort time and reduce the annual cooling and heating load. Finally, taking the typical apartment of Chongqing as an example, the annual cooling and heating load of the optimized design plan is decreased by 47.74% and the indoor thermal comfort time ratio is increased by 3.94%, which verify the feasibility and accuracy of the model.