A nonlinear dynamics bond graph model of gear pair system is established including the time-varying gear mesh stiffness, transmission error and tooth surface conditions. A capacitive component c is introduced to represent the loads and tooth elastic deformation, while the capacity parameter Kj is expressed by time-varying mesh stiffness, and flow Sf represents the influences of the gear transmission error on the system dynamic equations. Then the tooth surface friction bond graph model involving the relationship between the relative velocity and the direction of friction are presented. According to the causal relations and the power flow, the state-space equations of the gear bond graph model are obtained. Research results show that bond graph modeling method can solve the modeling problem of the gear nonlinear dynamics, and this paper is a kind of basis work of the complex mechanical system modeling and the development of bond graph modeling theory.