In order to solve the problems of large volume and high energy consumption of traditional permanent magnet top drive drilling system, hybrid permanent magnet vernier machine (HPMVM) is introduced into the top drive drilling system to form a new direct drive top drive system. In this paper, the coordinated control of flux observation and drive is studied. Firstly, the decoupling of permanent magnet flux linkage is realized in two-phase rotating coordinates, and a super twisting algorithm (STA) sliding mode permanent magnet flux observer is designed to realize the accurate observation of permanent magnet flux linkage. Then, the mapping relationship between the different speed ranges and the charging/demagnetizing state of the low coercivity permanent magnet is established, and the optimal torque control strategy combined with partition control is proposed. In the low-speed region, the optimal torque can be obtained when the motor is in saturation magnetization state; in the high-speed region, in order to reduce the excitation loss caused by the motor’s continuous adjustment of the permanent magnet magnetization state, the segmented magnetization control is adopted. Under the limitation of stator voltage and current, the Lagrange auxiliary function is constructed, and the optimal torque is obtained by calculating the extreme value of the function. Finally, compared with the traditional flux weakening control, the simulation shows that the STA sliding mode flux observer can quickly and accurately identify the permanent magnet flux, which can achieve the maximum torque output in the full speed domain, and reduce the copper consumption of the motor, thus improving the efficiency of the whole motor system.