MapReduce Job Scheduling in Hybrid Storage Modes
Author:
Affiliation:

Clc Number:

Fund Project:

  • Article
  • |
  • Figures
  • |
  • Metrics
  • |
  • Reference
  • |
  • Related
  • |
  • Cited by
  • |
  • Materials
  • |
  • Comments
    Abstract:

    In a heterogeneous Hadoop cluster scenario, the hybrid use of erasure codes and replica storage modes, as well as the real-time computing capability difference of server nodes lead to the low efficiency of MapReduce job processing. To deal with this problem, this study implements a scheduling strategy that dynamically adjusts MapReduce job assignment in multi-concurrent scenarios according to data storage situations and the real-time load of nodes. This strategy dynamically controls the concurrent amount of tasks of each node by modifying data storage location strategies in the current Hadoop framework, so as to achieve more balanced resource allocation among jobs when multiple jobs are concurrent. The experimental results show that the scheduling mode proposed in this study can shorten the job completion time by about 17% and effectively avoid the starvation phenomenon faced by some jobs compared with the two default job scheduling strategies of Hadoop.

    Reference
    Related
    Cited by
Get Citation

杨振宇,牛天洋,吕敏.混合存储模式下MapReduce作业调度.计算机系统应用,2023,32(3):70-85

Copy
Share
Article Metrics
  • Abstract:
  • PDF:
  • HTML:
  • Cited by:
History
  • Received:August 09,2022
  • Revised:September 15,2022
  • Adopted:
  • Online: December 09,2022
  • Published:
You are the firstVisitors
Copyright: Institute of Software, Chinese Academy of Sciences Beijing ICP No. 05046678-3
Address:4# South Fourth Street, Zhongguancun,Haidian, Beijing,Postal Code:100190
Phone:010-62661041 Fax: Email:csa (a) iscas.ac.cn
Technical Support:Beijing Qinyun Technology Development Co., Ltd.

Beijing Public Network Security No. 11040202500063