In order to study the variation law of mountain stability in karst area, considering the deterioration of back edge pipeline and sliding surface pipeline, as well as the weakening effect of water on structural plane. Taking the plane sliding of karst mountain as an example, two kinds of geological models and corresponding mechanical models are established, including unconnected and connected karst pipeline. Based on cusp catastrophe theory, deduced the stability coefficient (FOS) and critical stability coefficient (FOS*) of karst mountain under the action of karst pipeline flow, recommended the energy criterion of abrupt instability of mountain in karst area. The results showed that the displacement relationship between the equilibrium point and the critical instability point can indicate whether the mountain is unstable or not; there is one more factor about the stiffness coefficient k in the calculation formula of FOS and FOS* when the karst pipeline is unconnected, revealing that the karst pipeline has a certain effect on the mountain stability; FOS as an important factor affecting the mountain stability is related to the k and m. Through the case study of Jiwei Mountain landslide, it is proved that the energy criterion based on cusp catastrophe theory can better verify the stability of karst mountain with complex sliding surface structure, and can supply the results obtained by limit equilibrium method.