Image steganographic algorithm based on generative adversarial networks (GANs) can automatically learn embedding cost functions and enhance security, thus becoming a research hotspot in steganography. However, existing algorithms fail to adequately consider the mutual influence between neighbor-embedded pixels and the damage to image texture features caused by steganographic operations, which limits the security upper bound of these algorithms. To address the above issues, this study proposes an asymmetric cost learning algorithm based on adaptive texture perturbation minimization (AATPM). First, a neighbor-integrated reward correction strategy is designed to capture interactions between neighbor-embedded pixels by generating a reward correction coefficient matrix by considering pixel modifications within image sub-blocks. Second, a texture consistency loss function is proposed, which combines pixel-level and image-level texture feature perturbations to optimize embedding cost allocation through a dual-constraint mechanism, effectively maintaining texture feature consistency between cover and stego images. Comprehensive experiments show that AATPM outperforms other related steganographic algorithms in the performance of both handcrafted filter kernels and CNN-based steganalysis models.