Focusing on the issue that the correlation filter tracking algorithm under the condition of long-term occlusion or scale change has poor performance, the proposed algorithm makes the improvement based on the kernelized correlation filters tracking method. Firstly, the histogram of gradient and color-naming of target area are fused to construct training samples in order to improve the description of the target. Then, the scale is obtained by calculating the maximum response on the multi-scale image pyramid. Finally, the re-detection mechanism is introduced, and only when the response of the target is less than the threshold, the online random fern classifier is trained to re-detect objects. The obtained results of experiment demonstrate that the proposed algorithm is robust in the tracking of fast motion, heavy occlusion, out of view, and other complex scenes.