This paper introduces an approach to detect the quality defects of passports. The scumming defects of the passports are generally linear and produced in the course of passport printing. This phenomenon will have a severe impact on the overall quality of the passport. The chief difficulty to detect above defect is that the color of the scumming is close to the background texture of the image. Consequently, there is no relevant effectively detecting method. In order to detect the scumming defect, we first down-sample the image and analyze the multi-scale for reducing the influence of background pattern, and then employ a line segment detector to get these linear defects. The same line defect detection is conducted in both the samples and the template. We match the detected lines in each sample to eliminate the genuine lines while the non-matched lines are identified as defects. We also use the prior knowledge that linear defects are nearly vertical to improve the detection accuracy. Experiments show that the method of detecting linear defects is resultful. We deem that it is beneficial to control the passport quality and reduce the impact on the follow-up production of passports with this approach.