基于分组参数共享的跨层参数高效微调
作者:
作者单位:

作者简介:

通讯作者:

中图分类号:

基金项目:

陕西省自然科学基础研究面上项目(2023-JC-YB-825)


Cross-layer Parameter-efficient Fine-tuning Based on Grouped Parameter Sharing
Author:
Affiliation:

Fund Project:

  • 摘要
  • |
  • 图/表
  • |
  • 访问统计
  • |
  • 参考文献
  • |
  • 相似文献
  • |
  • 引证文献
  • |
  • 资源附件
  • |
  • 文章评论
    摘要:

    大型预训练模型在自然语言处理、计算机视觉等任务中展现出卓越的性能, 但在适配具体下游任务时仍需进行微调. 随着模型规模的持续增长, 现有参数高效微调(parameter-efficient fine-tuning, PEFT)方法在训练与推理阶段仍面临较高的计算与存储开销, 限制了其在资源受限场景中的应用. 针对上述问题, 本文提出一种基于分组参数共享的跨层参数高效微调方法, 在保持模型性能的同时, 进一步压缩可训练参数规模, 实现轻量化微调. 具体而言, 首先系统地分析参数共享机制在PEFT中的作用机理, 揭示跨层参数共享在压缩参数规模与保持模型性能之间取得平衡的内在原因. 其次, 从层间关联性出发, 引入中心化核对齐(centered kernel alignment, CKA)指标对层级表征相似性进行量化分析, 并据此设计合理的分组策略; 在此基础上, 设计跨层参数共享机制: 在功能相近的层组内共享一组参数, 并引入可学习缩放系数细粒度刻画层级差异, 实现轻量化微调与适配性能的协同优化. 实验结果表明, 在VTAB基准的19项下游任务中, 所提方法能大幅缩减参数量 (如可将LoRA参数缩减至原来的1/3), 在14个任务上取得最优性能, 其余任务与现有PEFT的表现相当, 进一步验证了本文方法的有效性和实用性.

    Abstract:

    Large pre-trained models have achieved excellent performance in tasks such as natural language processing and computer vision, but still require fine-tuning when adapting to specific downstream tasks. As model sizes continue to grow, existing parameter-efficient fine-tuning (PEFT) methods still incur substantial computational and storage costs during both training and inference, limiting their applicability in resource-constrained environments. To address this problem, this study proposes a cross-layer PEFT method based on grouped parameter sharing, which further compresses the number of trainable parameters while maintaining model performance, thus achieving lightweight fine-tuning. First, the role of parameter sharing in PEFT is systematically analyzed, revealing the underlying reasons for balancing parameter compression and maintaining model performance through cross-layer parameter sharing. Second, the centered kernel alignment (CKA) metric is introduced to quantitatively analyze the similarity of hierarchical representations, and a grouping strategy is designed based on inter-layer correlations. Finally, a cross-layer parameter sharing mechanism is designed, where a set of parameters is shared within functionally similar layer groups, and learnable scaling factors are introduced to characterize hierarchical differences, enabling lightweight fine-tuning while maintaining adaptation performance. Experimental results show that on 19 downstream tasks of the VTAB benchmark, the proposed method substantially reduces the number of parameters (e.g., LoRA parameters can be reduced to one-third of the original), achieves the best performance on 14 tasks, and performs comparably to existing PEFT methods on the remaining tasks, further verifying the effectiveness and practicality of the proposed method.

    参考文献
    相似文献
    引证文献
引用本文

张翔,于雪,刘美琪,张帆.基于分组参数共享的跨层参数高效微调.计算机系统应用,2026,35(8):294-306

复制
分享
相关视频

文章指标
  • 点击次数:
  • 下载次数:
  • HTML阅读次数:
  • 引用次数:
历史
  • 收稿日期:2025-12-31
  • 最后修改日期:2026-01-22
  • 录用日期:
  • 在线发布日期: 2026-06-25
  • 出版日期:
文章二维码
您是第位访问者
版权所有:中国科学院软件研究所 京ICP备05046678号-3
地址:北京市海淀区中关村南四街4号,邮政编码:100190
电话:010-62661041 传真: Email:csa@iscas.ac.cn
技术支持:北京勤云科技发展有限公司

京公网安备 11040202500063号