Safety fences play an important role in power construction sites. However, violations of crossing the fence are widespread, causing great safety hazards to the construction sites. To intelligently supervise, this study proposes a Faster RCNN-based detection method for crossing fence that combines the object detection and the ideas of frame difference method. The proposed method first obtains the information of the fence location and human keypoints by object detection from the captured frames in the video and then recognizes the violations at the construction site with the frame difference method. The experiment results show that the method can effectively detect violations of crossing fences at construction sites and meet the real-time requirements.