The failure mechanism of the ancient pagoda under compression was analyzed, based on the total and partial deformation characteristics of the pagoda model structure, and the evolution of the damage was calculated. Taking the first story of a pagoda as the prototype, three model specimens were generated by the use of old bricks at different scales: the original brick, the 1/4 and 1/8 scaled model brick. Compression tests were then carried out. The characteristics of the structural cracks developed during the loading process were observed. Numerical simulation was carried out. structural load, displacements, and strains were determined, and the stresses and failure modes of the three models were compared. The compression failure mechanisms and characteristics of the damage variation of the models were analyzed. The results indicate that the masonry failure of pagodas under compression occurs in three stages, the initial cracking along the mortar joint, the expansion and extension of the cracks, and the running through of the cracks. The cracking of the pagoda starts at the top four corners, gradually extends downwards, and finally runs through the entire structure with the increase in loading. Small sections of brick columns deformed, leading to structural failure owing to instability. In addition, horizontal deformation occurred along both the inside and the outside of the pagoda wall, and some bricks were cracked. The initial damage and stiffness of the model structure varied due to the different sizes of bricks. The cracking load and ultimate strength decreased as the unit block size decreased, while the strain followed an inverse trend.