Abstract:The attention mechanism in large language models faces severe computational and memory challenges when processing long sequences. Existing solutions are mainly developed from two perspectives: algorithmic optimization and sequence parallelism. At the algorithmic level, block sparse attention reduces computational complexity by introducing block sparsity into the mask matrix, among which Atrous sparsity and vertical-slash sparsity are two classical sparsity patterns. At the sequence-parallel level, ring attention enables efficient scaling across devices to support arbitrary long-context processing. However, when these two approaches are combined, ring attention still adopts a dense sequence partitioning strategy, which introduces substantial redundant communication and computation overhead and prevents the expected acceleration. To address this issue, customized sequence partitioning schemes and efficient distributed computing solutions are proposed for Atrous sparsity and vertical-slash sparsity. For Atrous sparsity, a grouped computing strategy is adopted, where data and devices are evenly divided into the same number of groups, and independent ring attention is executed within each group to significantly improve computational efficiency. For vertical-slash sparsity, a decomposition-based computing strategy is employed, in which the sparsity pattern is decomposed into slash sparsity and vertical sparsity subtasks. The slash sparsity subtask is processed through communication-free local computation, while the vertical sparsity subtask achieves performance acceleration by eliminating the computation and communication overhead of non-critical key-value blocks. Experimental results demonstrate that the proposed distributed optimization achieves 3–9 times performance improvement under Atrous sparsity and 3–10.9 times improvement under vertical-slash sparsity.