用于Hadoop2.x的MapReduce性能评估模型
作者:
作者单位:

作者简介:

通讯作者:

中图分类号:

基金项目:


MapReduce Performance Evaluation Model for Hadoop2.x
Author:
Affiliation:

Fund Project:

  • 摘要
  • |
  • 图/表
  • |
  • 访问统计
  • |
  • 参考文献
  • |
  • 相似文献
  • |
  • 引证文献
  • |
  • 增强出版
  • |
  • 文章评论
    摘要:

    基于MapReduce的程序被越来越多地应用于大型数据分析的应用中. Apache Hadoop是最常用的开源MapReduce模型之一. 程序运行时间的缩短对于MapReduce程序以及所有数据处理应用而言至关重要, 而能够准确估算MapReduce程序的执行时间是优化程序的重要环节. 本文定义了一个在Hadoop2.x版本中能够准确估算MapReduce作业负载执行时间的性能模型. 该模型包括一个优先级树模型与一个排队网络模型, 分别用于展示一个MapReduce作业中不同任务之间的依赖关系及MapReduce作业内的同步约束. 最后, 实验证明了该模型的可用性.

    Abstract:

    MapReduce-based systems are increasingly being used for large-scale data analysis applications. Apache Hadoop is one of the most common open-source implementations of such paradigm. Minimizing the execution time is vital for MapReduce as well as for all data-processing applications, and the accurate estimation of execution time is essential for optimization. In this study, the author created a MapReduce performance model for Hadoop2.x that can precisely estimate the execution time of workload in MapReduce. This model combines a precedence tree model that can capture dependencies between different tasks in one MapReduce job, and a queueing network model that can capture the intra-job synchronization constraints. Such an analytical performance model is a particularly attractive tool as it might provide reasonably accurate job response time at significantly lower cost than the simulation experiment of real data-analysis systems. Furthermore, a clear understanding of systematic job response time under different circumstances is key to making decisions in MapReduce workload management and resource capacity planning.

    参考文献
    相似文献
    引证文献
引用本文

吴岳.用于Hadoop2. x的MapReduce性能评估模型.计算机系统应用,2021,30(2):219-225

复制
分享
文章指标
  • 点击次数:
  • 下载次数:
  • HTML阅读次数:
  • 引用次数:
历史
  • 收稿日期:2020-06-29
  • 最后修改日期:2020-07-27
  • 录用日期:
  • 在线发布日期: 2021-01-29
  • 出版日期:
您是第位访问者
版权所有:中国科学院软件研究所 京ICP备05046678号-3
地址:北京海淀区中关村南四街4号 中科院软件园区 7号楼305房间,邮政编码:100190
电话:010-62661041 传真: Email:csa (a) iscas.ac.cn
技术支持:北京勤云科技发展有限公司

京公网安备 11040202500063号