Planetary Roller Screw Mechanism (PRSM) is a new type of heavy duty precision transmission device, which is transformed from rotating motion to linear motion. The machining quality of the thread tooth directly determines its transmission accuracy and load-bearing performance. Aiming at the problem of non-complete short arc detection and evaluation of PRSM roller tooth profile with small pitch, a roller tooth profile curve fitting method based on nonlinear interior point algorithm was proposed by using high-precision CMM for non-contact detection of roller axial section tooth profile, and the fitting accuracy of roller tooth profile curve was verified. The evaluation model and evaluation method of roller thread tooth profile deviation were established for the first time, the evaluation chart of roller tooth profile deviation based on the model was drawn, and the distribution of roller tooth profile deviation was analyzed and summarized. The proposed PRSM roller tooth profile detection and evaluation method with small pitch has a fitting accuracy of 1.5E-4, the fitting accuracy of tooth profile radius is controlled at 6 μm, and the fitting accuracy of tooth profile center z0 coordinate is controlled at 9.8 μm. It can quantitatively evaluate the shape accuracy and position accuracy of roller tooth profile, and has practical feasibility in engineering practice.