To achieve low-cost and high-precision, an indoor location system with simple visual labels is designed by comprehensively considering the requirement and computation complexity. Only the color and shape features are used for label detection, by which both detection complexity and data storage requirement are reduced. To deal with the nonunique solution problem caused by the simple label features, a fast query and matching method is further presented by using the view field of camera and azimuth of label. Then based on the relationship analysis between the distribution characteristics of labels and location error, a pose and position estimation method using the weighted least square algorithm is designed and cooperated with the interactive algorithm by the designed switching strategy. A good balance has been made between the algorithm complexity and location accuracy by the presented techniques. The simulation and experiment results show that the proposed method can deal with the singular problem of the overdetermined equations effectively and attenuate the negative effect of bad label groups. Compared with the ultra-wide band technology, the location error is reduced more than 62%.