The soft manipulator possesses dexterity and flexibility, ensuring safe interaction with the environment while accurately tracking position and posture. It has emerged as a prominent area of research in recent years. However, because the material deformation of the soft manipulator is nonlinear, its kinematic modeling parameters are numerous and it is difficult to obtain accurate values, these difficulties hinder the realization of kinematic control for the soft manipulator. To address the uncertainty of the soft manipulator, this paper proposes a new hand-eye visual servoing method driven by historical data, building upon the current visual servoing techniques. This method integrates a controller based on the random forest algorithm to accomplish the control tasks of the manipulator. By clustering historical data, an inverse mapping of the driving state of the soft manipulator and image characteristics is established using the random forest regression model. The system input variables are predicted quickly without the need to solve any parameters of the manipulator and camera. The experimental results show that the proposed method can better achieve the expected control objectives.