Abstract:Large language model pre-training typically relies on large-scale graphics processing unit (GPU) clusters. Existing pipeline parallelism methods are mostly based on assumptions of homogeneity and struggle to adapt to heterogeneous environments with significant differences in computing power, memory capacity, and communication. To address this issue, this study formulates heterogeneous pipeline orchestration as a joint optimization problem of model partitioning, topology construction, and task scheduling, and proposes adaptive heterogeneous pipeline parallelism (AHPipe). This method obtains device computation, memory, and communication costs through performance analysis, uses simulated annealing to search for model partitioning and pipeline topology schemes, and employs adaptive task scheduling to reduce pipeline bubbles under memory constraints. Experimental results on a heterogeneous GPU cluster consisting of NVIDIA V100, H20, RTX-4090, and RTX-5090 GPUs and covering GPT-3 models from 760M to 13B parameters show that the proposed method improves training throughput by up to 39.2% over representative existing methods in heterogeneous scenarios and can generate feasible training schemes under memory-constrained settings.