To achieve a low-cost, high-precision indoor location system, this study designs a method using simple visual labels while balancing computation complexity and practical requirements. Only color and shape features are used for label detection, minimizing both detection complexity and data storage needs. To deal with the issue of nonunique solutions caused by simplified label features, a rapid query and matching method is proposed by incorporating the camera’s field of view and the label's azimuth. Furthermore, a pose and position estimation method using a weighted least square algorithm is developed. This method is integrated with an interactive algorithm guided by a designed switching strategy. These techniques strike an effective balance between algorithm complexity and location accuracy. Simulation and experimental results show that the proposed method effectively resolves singularity issues in overdetermined equations and attenuates the negative effects of poorly distributed label groups. Compared with ultra-wideband technology, the proposed approach reduces location error by more than 62%.