Focusing on the problem of computer operation without hand for the disabled people, a new method of mouse control based on visual analysis of the face poses is proposed. First, the face images from CCD camera are obtained and the face areas are located based on AdaBoost algorithms with Haar features. Then, 5 feature points including 2 eye corners, 2 mouth corners and nose tip are detected with image analysis. After that, with help of the feature points in a predefined frontal face image, the face poses are estimated in the real-time face images. Finally, the 3 deflection angles are adopted to define human computer interactive information, which can be used to determine positions and operations of the mouse in the human computer interactive interface. The experimental results demonstrate the efficiency of our method that the face poses can be estimated accurately as well as the operation information to the mouse be achieved rapidly. It is valuable in human computer interaction for the disabled people.