This paper proposes a novel object tracking approach via graph-based manifold ranking to handle the model drift problem in the tracking-by-detection framework. The proposed approach can suppress the effects of background information caused by object deformation, scale variation and occlusion in object tracking. First, we partition the target bounding box into non-overlapping image patches, and take these image patches as graph nodes to construct k-regular graph, in which the edge weight between two neighbor nodes are measured by the distance of their low-level features. Second, we assign each patch with a weight describing the importance in representing the object, and compute it in a semi-supervised way. In particular, we initialize some patches as object patches with the weights 1, and remaining patches with the weights 0. The graph-based manifold ranking is then performed to obtain the weights of all patches. Moreover, we propose to determine the optimal scale based on multi-scale feature pyramid to address scale adaptation while improving the quality of initial patches in object tracking. Finally, we concatenate all weighted patch descriptors into a vector to represent the bounding box feature, and then integrate it into structure output (Struck) algorithm to carry out object tracking. Experimental results on several public video sequences suggest that the proposed method significantly outperforms other tracking methods.