Prostate cancer is a common malignant tumor among men, and the automatic anomaly detection of its MRI images is crucial for improving diagnostic efficiency and reducing doctors’ workload. To address the problem that existing methods rely on large amounts of labeled data and have limited ability to detect subtle lesions, this paper proposes an unsupervised prostate cancer anomaly detection method based on ensemble autoencoders (EAE) and residual-based pseudo-anomaly generation. Specifically, an EAE with centered kernel alignment (CKA) loss is first introduced to suppress feature redundancy and produce diverse reconstruction results. Secondly, by integrating the consensus high-response regions from the multi-branch EAE, anatomically constrained perturbations are injected in the condition of no manual labels to generate pseudo-anomaly samples that more closely resemble real lesions, all without manual labels. Furthermore, a frequency-domain contrastive loss is introduced in the detection network to enhance the ability to distinguish between normal and anomaly samples in the frequency domain space. Finally, a two-stage process is adopted to maintain consistency between training and inference, thus improving model robustness. Extensive experiments show that the proposed method yields superior detection accuracy and generalization ability on public prostate MRI and brain MRI datasets. Specifically, on datasets such as PICAI, PMU and BTM, the image-level AUC reaches 84.00%, 89.20%, and 89.50% respectively, and the AP values are 82.5%, 87.00%, and 88.80% respectively, outperforming current state-of-the-art methods.