基于SimCSE框架融合预训练模型层级特征的文本匹配
作者:
作者单位:

作者简介:

通讯作者:

中图分类号:

基金项目:

国家重点研发计划(2019YFB1405303); 北京市属高等学校优秀青年人才培育计划(BPHR202203233); 国家自然科学基金面上项目(72174018)


Text Matching Based on SimCSE Framework Fused with Pre-trained Model Internal Hierarchical Features
Author:
Affiliation:

Fund Project:

  • 摘要
  • |
  • 图/表
  • |
  • 访问统计
  • |
  • 参考文献
  • |
  • 相似文献
  • |
  • 引证文献
  • |
  • 增强出版
  • |
  • 文章评论
    摘要:

    SimCSE框架仅使用分类令牌[CLS]token作为文本向量, 同时忽略基座模型内层级信息, 导致对基座模型输出语义特征提取不充分. 本文基于SimCSE框架提出一种融合预训练模型层级特征方法SimCSE-HFF (SimCSE with hierarchical feature fusion, SimCSE-HFF). SimCSE-HFF基于双路并行网络, 使用短路径和长路径强化特征学习, 短路径使用卷积神经网络学习文本局部特征并进行降维, 长路径使用双向门控循环神经网络学习深度语义信息, 同时在长路径中利用自编码器融合基座模型内部其他层特征, 解决模型对输出特征提取不充分的问题. 在STS-B的中文与英文数据集上, SimCSE-HFF方法效果在语义相似度SpearmanPearson相关性指标上优于传统方法, 在不同预训练模型上均得到提升; 在下游任务检索问答上也优于SimCSE框架, 具有更优秀的通用性.

    Abstract:

    The simple contrastive learning of sentence embedding (SimCSE) framework only uses the classification [CLS]tokens as text vectors, and it also neglects the hierarchical information within the base model, which results in insufficient extraction of semantic features from the base model output. Based on the SimCSE framework, this study proposes a method that fuses hierarchical features of pre-trained models, SimCSE with hierarchical feature fusion (SimCSE-HFF). SimCSE-HFF is based on a dual-path parallel network, using short and long paths to strengthen feature learning. The short path uses a convolutional neural network to learn local text features and perform dimensionality reduction, while the long path uses a bidirectional gated recurrent neural network to learn deep semantic information. Additionally, in the long path, an autoencoder is used to fuse features from other layers within the base model, solving the problem of insufficient extraction of output features by the model. On the Chinese and English datasets of spring tools suite-bundle (STS-B), the SimCSE-HFF method outperforms traditional methods in terms of semantic similarity Spearman and Pearson correlation metrics, showing improvements on different pre-trained models. Additionally, it also outperforms the SimCSE framework in downstream task retrieval-based question answering, demonstrating better versatility.

    参考文献
    相似文献
    引证文献
引用本文

盛成城,陈进东,张健.基于SimCSE框架融合预训练模型层级特征的文本匹配.计算机系统应用,,():1-9

复制
分享
文章指标
  • 点击次数:
  • 下载次数:
  • HTML阅读次数:
  • 引用次数:
历史
  • 收稿日期:2024-01-10
  • 最后修改日期:2024-02-07
  • 录用日期:
  • 在线发布日期: 2024-06-05
  • 出版日期:
您是第位访问者
版权所有:中国科学院软件研究所 京ICP备05046678号-3
地址:北京海淀区中关村南四街4号 中科院软件园区 7号楼305房间,邮政编码:100190
电话:010-62661041 传真: Email:csa (a) iscas.ac.cn
技术支持:北京勤云科技发展有限公司

京公网安备 11040202500063号