Abstract:Multi-agent cooperative reinforcement learning (MACRL) algorithms are widely applied in various scenarios; however, significant deficiencies in information utilization remain, including insufficient filtering of redundant information, the absence of coordination mechanisms among multi-dimensional information, and inadequate consideration of the influence of other agent’ behaviors. To address these issues, this study proposes a sequential-evaluation-driven Shapley value Q-learning model with internal and external embedding enhancement. Inspired by reciprocal determinism in social learning theory, which states that individual factors, the external environment, and behaviors jointly determine outcomes, the proposed model consists of an internal embedding module, an external embedding module, and a sequential-evaluation-driven Shapley value Q-learning module, which are responsible for learning individual factor representations, external environment representations, and action value evaluation, respectively. Specifically, key external information is adaptively captured through the external embedding module, while local observations and historical experience learnings are optimized by the internal embedding module. A fusion module is further employed to model the synergistic effects between internal and external information. To address the credit assignment problem, a sequential Shapley value Q-learning method is proposed. Marginal contribution estimation is optimized based on marginal benefit decomposition theory, and the Shapley value computation is reformulated as a sequence modeling problem to quantify the contribution of other agents’ decisions to the current agent’s action selection. Experimental results show that the proposed algorithm achieves an average win rate of 0.85 across eight complex StarCraft II environments, reaching state-of-the-art (SOTA) performance.