In modern securities investment theories and methods, the optimal risk portfolio is one of the most important core concepts, but the existing literature lacks rigorous mathematical expression for this theory and its related conclusions. In order to solve this problem, we have studied the mathematical mechanism related to the formation of the concept of the optimal risk portfolio in portfolio selection and made strict logical deduction and mathematical modeling analysis. First of all, the mathematical analysis method was used to construct the corresponding relationship, that is, to establish one-to-one correspondence between the risk, the return of portfolio and the number pairs of plane coordinate system, and the algebraic method was employed to define an order relationship among these number pairs. From the perspective of analytic geometry and through the analysis of the intersection coordinates of quadric clusters, the mathematical equations that meet the relevant conditions of portfolio theory were constructed on the basis of the relevant properties of conic in two-dimensional plane, and then mathematical derivation and solution were carried out. Finally, by analyzing the mathematical model of the investment opportunity set of the binary portfolio, the mathematical expression of the optimal risk portfolio in the binary portfolio was determined. And this method can be extended to the case of multiple portfolio.