本文已被:浏览 997次 下载 1912次
Received:November 07, 2022 Revised:December 10, 2022
Received:November 07, 2022 Revised:December 10, 2022
中文摘要: 针对视频描述生成的文本质量不高与不够新颖的问题, 本文提出一种基于特征强化与文本知识补充的编解码模型. 在编码阶段, 该模型通过局部与全局特征强化增强模型对视频中静态物体的细粒度特征提取, 提高了对物体相似语义的分辨, 并融合视觉语义与视频特征于长短期记忆网络(long short-term memory, LSTM); 在解码阶段, 为挖掘视频中不易被机器发现的隐含信息, 截取视频部分帧并检测其中视觉目标, 利用得到的视觉目标从外部知识语库提取知识用来补充描述文本的生成, 以此产生出更新颖更自然的文本描述. 在MSVD与MSR-VTT数据集上的实验结果表明, 本文方法展现出良好的性能, 并且生成的内容信息在一定程度上能够表现出新颖的隐含信息.
Abstract:As texts generated by video descriptions are of low quality and not novel, this study proposes a codec model based on feature reinforcement and text knowledge supplementation. In the coding stage, the model enhances the fine-grained feature extraction of static objects in a video by strengthening local and global features, thus improving the resolution of similar semantics of objects. Then, it integrates visual semantics and video features into a long short-term memory (LSTM) network. In the decoding stage, to mine the hidden information that can hardly be discovered by machines in the video, the model intercepts partial video frames and detects the visual goals in them. Then, the obtained visual goals are used to extract knowledge from the external knowledge base to supplement the generation of descriptive texts and thus produce more novel and natural text descriptions. The experimental results on datasets MSVD and MSR-VTT demonstrate that the proposed method shows good performance, and the generated content can show novel implicit information to a certain extent.
keywords: video description codec model feature reinforcement visual goals knowledge supplementation artificial intelligence natural language processing (NLP)
文章编号: 中图分类号: 文献标志码:
基金项目:陕西省科技计划重点项目(2017ZDCXL-GY-05-03)
引用文本:
王林,白云帆.基于特征强化与知识补充的视频描述方法.计算机系统应用,2023,32(5):273-282
WANG Lin,BAI Yun-Fan.Video Description Method Combining Feature Reinforcement and Knowledge Supplementation.COMPUTER SYSTEMS APPLICATIONS,2023,32(5):273-282
王林,白云帆.基于特征强化与知识补充的视频描述方法.计算机系统应用,2023,32(5):273-282
WANG Lin,BAI Yun-Fan.Video Description Method Combining Feature Reinforcement and Knowledge Supplementation.COMPUTER SYSTEMS APPLICATIONS,2023,32(5):273-282

