Broken needles are frequently seen in the production process of clothing and shoe factories. This study proposes a detection system of metal foreign bodies in sole of shoes based on deep learning since those residual bodies such as broken needles in shoes will threaten people’s safety. Firstly, shoes are put on a conveyor belt in turn and sent to a needle detector, and the images are collected by X-ray irradiation. After that, the images are preprocessed to highlight the small metal foreign bodies. Finally, metal foreign bodies and their positions are detected with a deep learning network model. Experimental results show that preprocessing images and fine-tuning the label box can make metal foreign bodies clearer, and the average precision of the model is 97.6%. It proves that the model can effectively detect the metal foreign bodies with different shapes left in footwear, presenting great commercial potential.