Active vibration control uses an active anti-force from secondary path to suppress or attenuate vibration and noises. One of the wildly-used control algorithms is filtered-x least mean square(FxLMS) adaptive algorithm. Based on structure of FxLMS algorithm，Level-2 S function is used to build a new FxLMS blocks in Matlab/Simulink，which is applied to offline vibration active control system simulation. On the condition of convergence，performance analysis is carried out by adjusting interior parameters to test the control algorithm block. Finally，the custom FxLMS block is downloaded to dSPACE as controller，and used in hardware-in-the-loop simulation of active vibration control on a gear transmission system. Results verified the custom FxLMS block’s feasibility built by Level-2 S function and control algorithm’s efficiency.