Phosphogypsum, as one of the industrial solid wastes, can enhance the soil strength but also increase the brittleness while the fiber reinforcement technique can improve the soil plasticity. The synergy between the two materials (phosphogypsum and fiber) can play a complementary role in improving the mechanical properties of soils. The loess from Yangling, Shaanxi Province, was used to explore the optimal mix ratio of phosphogypsum in the loess by unconfined compressive strength (UCS) tests. Then, the optimal mix ratio of fiber in the mixture of phosphogypsum and loess was investigated by the direct shear test based on the mixture ratio above. Finally, the microscopic mechanisms of loess improvement with phosphogypsum and polypropylene fibers were examined through scanning electron microscopy (SEM). The results showed that the loess UCS increases first and then decreases with an increase in the phosphogypsum mass when the phosphogypsum is mixed with the loess for reinforcement, and the optimal mix mass ratio of phosphogypsum in the loess is 12% in this study. The polypropylene fibers were then mixed into the mixture of phosphogypsum and loess based on the optimal mix mass ratio of phosphogypsum in the loess. It can be found that the peak strength, cohesion, and residual strength of the mixture sample increase first and then decrease, while the internal friction angle increases, with an increase in the polypropylene fiber content. The microscopic test results also indicated that the phosphogypsum-improved loess and phosphogypsum-polypropylene fiber-improved loess both significantly enhance the cementation and fill the voids among soil particles. Polypropylene fibers in loess can form fiber clumps to wrap the soil particles, thereby providing tensile stress and improving the plasticity for the soil sample.