In order to effectively remove salt and pepper noise and protect image details, this paper proposes an improved algorithm based on the extreme value of salt and pepper noise filtering. Firstly, the algorithm detects noise, each pixel is classified to be possible noise pixel and signal pixel, and the value of possible noise pixels is near 0 and 255, and does not constitute 5 pixels or more straight points, the remaining points is as the signal point. Then, the noise filtering is taken, in order to protect the image edge details, or thread, signal pixel without any treatment, and the noise pixels using the gradient method for processing. The experimental results show that the new algorithm not only can effectively remove salt and pepper noise, and effect made in detail preservation is superior to traditional median filtering algorithm.