Abstract:Large language models (LLMs) demonstrate superior performance due to their advanced reasoning capabilities and extensive training on massive datasets. However, their large scale leads to high computational costs and deployment challenges. Knowledge distillation, particularly chain-of-thought distillation (CoTD), effectively compresses models while preserving reasoning ability by transferring the teacher model’s reasoning process to the student model. Nevertheless, traditional CoTD suffers from two key limitations: insufficient step-level supervision, where using only the complete reasoning chains as supervision ignores intermediate decisions and hinders the student model from fully mastering the teacher’s reasoning logic; and exposure bias, where training depends on teacher-provided reasoning chains while inference relies on the student’s self-generated outputs, causing a training-inference distribution mismatch that accumulates errors and degrades performance. To address these issues, a progressive CoTD method is proposed. The teacher’s reasoning chain is decomposed into fine-grained semantic reasoning units to provide step-level supervision, while a progressive scheduling approach is introduced to gradually transition from teacher-generated inputs to student self-generated inputs, thereby bridging the distribution gap. A flexible scheduling function is designed to ensure a smooth transition, optimized by adjusting the maximum student input probability and growth dynamics. Experimental results show that the proposed method outperforms several existing CoTD approaches on multiple datasets including StrategyQA and ARC, demonstrating enhanced student model performance.