Abstract:In reinforcement learning, experience replay is a key technique to improve the efficiency of policy learning. However, how to effectively select and reuse experiences remains a challenging issue. Using experiences with high gradient coherence in experience replay can better enhance policy stability and reduce the noise brought by high temporal difference (TD) errors. Based on this, gradient coherence optimized experience replay (GCER) is proposed, an algorithm that not only relies on traditional TD errors but also incorporates gradient coherence to select experiences. This method can filter out the most valuable experiences for policy improvement while avoiding unstable updates due to drastic changes in gradient direction. Experimental results show that the experience selection mechanism combining gradient coherence and TD errors significantly enhances the learning efficiency and stability of policies, providing a new perspective and direction for experience sampling strategies in reinforcement learning.