This paper develops the improved geometric constraint model (IGCM) based on the 3D block model. The variable-slope excavation cones of each block have been abstracted to collections of nodes and directed edges, and the optimization method has been introduced using the set theory. The IGCM consists of the optimized collection nodes and directed edges. The algorithm based on IGCM has been actualized in DIMINE digital mine system, and is applied to an actual open pit mine boundary optimization engineering. The results shows that IGCM can accurately simulate variable slope angles in any azimuth and elevation changes, and adapt to all open pit mine boundary optimum problems in any complex geological conditions, calculation speed and accurate. It provides a new and effective method for the variable-slope open pit mine boundary optimization problem in complex geological conditions.