It is important to find what we need from the massive video, thus the research of key frame extraction algorithm is becoming more and more hot. First, the paper present the algorithm about extraction key frames from the surveillance video. Then camera calibration and vanishing point is used to analyse and calculate the height of human body. The result showed that semantic distance based on color histogram is more suitable for extracting key frame from the surveillance video. Meanwhile this algorithm of human body height is very feasible and accurate. So the algorithm has practical value.