In order to improve the abrasion resistance of concrete, the effects of granular, acicular and powdered rubbers on the abrasion performance of concrete were studied. Based on granular rubber concrete, basalt fiber was added to study the abrasion performance of concrete when fiber-rubber composite was used. The microstructure was observed by SEM and the abrasion enhancement mechanism was analyzed. The fractal dimension was used to characterize the morphology change of concrete abrasion process, and the relationship between fractal dimension and abrasion degree was discussed. The results show that when 15% of granular rubber, acicular rubber and powdered rubber are added respectively, the abrasion resistance of concrete increases by 140.24%, 157.96% and 83.88%. When the basalt fiber content is 0.1%, the abrasion resistance of granular rubber concrete increases by 11.63%. Fiber and rubber can play a role at different levels and jointly enhance the abrasion resistance. The degree of concrete abrasion can be characterized by the fractal dimension. With the increase of time, the more serious the concrete abrasion is, the larger the fractal dimension is. The relationship between fractal dimension and mass loss rate and volume loss rate is obtained, and the curve equation is established.