In the paper we look into the building of the rule of law in China from historical perspective. By reviewing the history of the Chinese legal system, as a whole the Chinese legal system is divided into five periods of rule of ritual and so on. We analyze their characteristics. From analyzing legal system events of ancient and modern, we present that the progress of the rule of law often needs a high price, the rule of law in China should learn lessons and avoid detours. Due to historical inertia and practical issues, the rule of law in China still has a long way to go and can’t be finished in a short time. If we want to achieve the final success of the rule of law, we must establish national self-confidence and willpower, train law-abiding habits and form our own system of legal philosophy.