A Dynamic MapReduce Scheduler for Heterogeneous Workloads

Chao Tian, Haojie Zhou , Yongqiang He,Li Zha

簡報人：碩資工一甲 董耀文

Outline

Background Question? So! Related work MapReduce procedure analysis MR-Predict Schedule policys Evaluation Conclusion

Background

As the Internet scale keeps growing up, enormous data needs to be processed in many Internet Service Providers.

MapReduce framework is now becoming a leading example solution, it’s designed for building large commodity cluster, which consist of thousands of nodes by using commodity hardware.

Background

The performance of a parallel system like MapReduce system closely ties to its task scheduler.

Current scheduler in Hadoop uses a single queue for scheduling jobs with a FCFS method.

Yahoo’s capacity scheduler as well as Facebook’s fair scheduler uses multiple queues for allocation differnet resource in the cluster.

Background

In practical, different kinds of jobs often simultaneously run in the data center. These different jobs make different workloads on the cluster, including the I/O-bound and CPU-bound workloads.

Background

The characters of workloads are not aware by Hadoop's scheduler which prefers to simultaneously run map tasks from the same job on the top of queue.

This may reduce the throughput of the whole system which seriously influences the productivity of data center, because tasks from the same job always have the same character.

Question

How to improve the hardware utilization rate when different kinds of workloads run on the clusters in MapReduce framework?

SO!

They design a new triple-queue scheduler which consist of a workload predict mechanism MR-Predict and three different queues (CPU-bound queue, I/O-bound queue and wait queue).

They classify MapReduce workloads into three types, and their workload predict mechanism automatically predicts the class of a new coming job based on this classification.

Jobs in the CPU- bound queue or I/O-bound queue are assigned separately to parallel different type of workloads.

Their experiments show that can Approach could increase the system throughput up to 30%

Related work

Scheduling algorithms in parallel system [11,…]

Applications have different workloads large computation and I/O requirements [10].

How I/O-bound jobs affect system performance[6].

A gang schedule algorithm which parallel the CPU- bound jobs and IO-bound jobs to increasing the utilization of hardware[7].

Related work

The schedule problem in MapReduce attracted many attentions[2,10].

Yahoo and Facebook designed schedulers of Hadoop as capacity scheduler [4] and Fair scheduler [5].

MapReduce procedure analysis

Map-shuffle phase1. Init input data2. Compute map

task3. Store ouput result

to local disk4. Shuffle map tasks

result data out5. Shuffle reduce

input data in

MapReduce procedure analysis

Reduce-Compute phase1. tasks run the

application logic

MR-Predict

Schedule policys

Schedule policys

Evaluation

Environment 6 node connect gigabyte Etherent. DELL1950

CPU: 2 Quard Core 2.0GHz Memory: 4GB Disk: 2 SATA disk

Input data: 15GB map slots & reduce slot: 8 DIOR: 31.2 MB/s (without reduce phase in

Hadoop)

Evaluation

Resource utilizations

TeraSort: Total order sort (sequential I/O )benchmark

8 ( 64MB + 64 MB ) / 8 >= 31.2 MB/s

Evaluation

Resource utilizations

Grep-Count: use [.]* as the regular expression.

8 ( 64MB + 1MB + 1MB + SID ) / 92 >= 31.2 MB/s

Evaluation

Resource utilizations

WordCount: It splits the input text into words, shuffles every word in map phase and counts its occupation number in reduce phase.

8 ( 64MB + 64 MB + 64MB + SID ) / 35 >= 31.2 MB/s

Evaluation

Triple queue scheduler experiments Every job runs five times & total 15 jobs will run

Conclusion

Scheduler correctly distributes jobs into different queues in most situations.

Triple Queue Scheduler could increase the map tasks throughput 30% save the makespan 20%

Thank you for listening.