The current research on KL divergence-based fuzzy clustering segmentation faces two core challenges. The first is how to effectively balance the noise tolerance and computational efficiency of the algorithms to meet the requirements for real-time performance. The second is how to avoid the local optimum caused by non-convex objective functions and improve the accuracy and stability of complex images. To this end, this study proposes a fast fuzzy clustering image segmentation algorithm that integrates intra-class and inter-class distances and KL divergence. Firstly, the classical idea of solely minimizing the intra-class distance is abandoned, and a new objective metric based on the difference of minimizing the intra-class distance and maximizing the inter-class distance is constructed, which ensures the minimization of intra-class distance and maximization of inter-class distance. Additionally, this guarantees that the sample points can accurately find its corresponding class during classification. Secondly, KL divergence is innovatively combined with histograms of images. On the one hand, KL divergence is employed to enhance the robustness for noise and non-uniform data. On the other hand, the utilization of the histograms can greatly reduce the data amount of algorithm iteration, which improves the local consistency and ensures the algorithms’ efficiency. As a result, the difficulty of existing methods in balancing “robustness”, “accuracy” and “real-time performance” is solved to make the algorithm more applicable to fields such as medicine, intelligent driving, and robot navigation. A large number of various kinds of image segmentation tests show that the proposed intra-class and inter-class KL divergence-based fuzzy C-means clustering algorithm is effective, especially for segmenting large images with big noise. The algorithm can not only remove noises but also satisfy the requirements of real-time segmentation.