In the process of target tracking, the accuracy of tracker for single image source is not high enough and the target would be inclined to be lost when covered partially. A method of fusing the features of infrared image and visible light image is proposed in this study. First, the color information of the visible light image is extracted as a parameter in the target model, and the gray level information of the infrared image is taken as the other parameter. According to the two parameters, the target positions and its subgraphs can be acquired respectively. Then the corresponding Bhattacharyya coefficients are calculated by the anterior target subgraphs and the target models. The weights of the respective coefficients can be calculated on the basis of the weighting function. Finally, the target that is weighted with the Mean Shift algorithm could be tracked. This method makes full use of the advantages of infrared images and visible light images, improves the accuracy of tracker, and has solved the problem that the target is likely to be lost when partially covered.