Planetary Roller Screw Mechanism (PRSM) is a new type of heavy duty precision transmission device, which can realize the transformation from rotating motion to linear motion and vise versa. The machining quality of the thread tooth directly determines its transmission accuracy and load-bearing performance. To address the problem of non-complete short arc detection and evaluation of PRSM roller tooth profile with small pitch, a roller tooth profile curve fitting method based on nonlinear interior point algorithm was proposed by using high-precision CMM for non-contact detection of roller axial section tooth profile, and the fitting accuracy of roller tooth profile curve was verified. The evaluation model and evaluation method of roller thread tooth profile deviation were established for the first time. The evaluation chart of roller tooth profile deviation based on the model was drawn, and the distribution of roller tooth profile deviation was analyzed and summarized. The proposed PRSM roller tooth profile detection and evaluation method with small pitch had a fitting accuracy of 1.5×10^-4. The fitting accuracy of tooth profile radius and tooth profile center z₀ coordinate could be controlled at 6 μm and 9.8 μm, respectively. It can quantitatively evaluate the shape accuracy and position accuracy of roller tooth profile, and has practical feasibility in engineering practice.