دانلود رایگان مقاله تخصیص منابع چند لایه برای محاسبات داده های فشرده

Abstract

As distributed computing systems are used more widely, driven by trends such as ‘big data’ and cloud computing, they are being used for an increasingly wide range of applications. With this massive increase in application heterogeneity, the ability to have a general purpose resource management technique that performs well in heterogeneous environments is becoming increasingly important. In this paper, we present Multi-Tier Resource Allocation (MTRA) as a novel fine-grained resource management technique for distributed systems. The core idea is based on allocating resources to individual tasks in a tiered or layered approach. To account for heterogeneity, we propose a dynamic resource allocation method that adjusts resource allocations to individual tasks on a cluster node based on resource utilisation levels. We demonstrate the efficacy of this technique in a data-intensive computing environment, MapReduce data processing framework in Hadoop YARN. Our results demonstrate that MTRA is an effective general purpose resource management technique particularly for data-intensive computing environments. On a range of MapReduce benchmarks in a Hadoop YARN environment, our MTRA technique improves performance by up to 18%. In a Facebook workload model it improves job execution times by 10% on average, and up to 56% for individual jobs.

6. Conclusion

In this paper, we have presented a novel resource allocation technique (MTRA) that dynamically adjusts resource allocations to individual tasks. By introducing a third resource allocation tier, MTRA adjusts resource allocations in a fine-grained manner based on resource usage levels on each cluster node. It is schedulerindependent, resulting in increased scalability, and also allows finer-grained control than would be possible with a scheduler based solution. In our evaluation, we have implemented MTRA in Hadoop YARN. We have observed performance improvements of up to 18% compared to Hadoop YARN for individual MapReduce benchmarks. For a multi-job workload model based on a Facebook production cluster workload, we have shown that MTRA reduces individual job execution times by 10% on average and up to 56% for individual applications. We conclude that our novel dynamic resource allocation technique succeeds in meeting the need for a general-purpose scheduling technique which can account for application heterogeneity