The dynamic behavior and energy evolution of rockburst at the moment of sudden instability are the key to reveal rockburst formation mechanism. Based on the instability theory and stiffness theory, a combination model consists of rockburst body and its surrounding rock was established using the three-dimension discrete element method (DEM). With it the energy evolution process of the system is visualized through secondary development and the ejection phenomenon of rockburst is simulated successfully. Based on the numerical simulation, the time sequence characteristics and energy evolution laws of instability of the system are studied. The results show that the rockburst rock body reaches the peak stress slightly earlier than the surrounding rock body. The rupture and softening of the rockburst body after the peak stress results in the unloading of the surrounding rock body, and causes its stress changes from increase to decreases. The surrounding rock mass then has a rapid rebound deformation to the rockburst body, and the elastic energy stored in the surrounding rock body converges to the rockburst body. In this process, the positive feedback regulation mechanism formed by the interaction of the two subsystems accelerates the fracture processes and finally leads to dynamic instability. The research results can provide theoretical reference and technical support for further revealing the rockburst mechanism.