YARN Resource Allocation When Memory is Very Small

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

YARN Resource Allocation When Memory is Very Small

Nico Pappagianis
How does YARN decide if remaining memory is enough AM allocation

Hi all,

Somewhat of a newb here so hopefully this question isn't too trivial.

I want to know how YARN decides if the remaining memory after AM resource allocation is "enough". Here's a breakdown of what's going on in my particular case.

We have a YARN queue that has ~1.7GB memory allocated to it. We have a parent MR job that spawns multiple child jobs. The AM container memory for the parent job is configured to 1GB. Once the AM container gets allocated its 1GB there is a remaining ~0.7GB memory available. In my case that 0.7GB is enough for the child jobs to succeed. 

My question is, what if that remaining amount wasn't 0.7GB but something much less, like 0.1GB, or 0.0001GB? How does YARN handle those cases - will it allocate any remaining resources to the child jobs? Will there be a ton of thrashing if there is only a small amount of memory available?

Thanks for any help, or a pointer in the right direction!
Reply | Threaded
Open this post in threaded view
|

Re: YARN Resource Allocation When Memory is Very Small

Ravi Prakash-3
Hi Nico!

The RM is configured with a minimum allocation. Take a look at "yarn.scheduler.minimum-allocation-mb" . You can also read through code here: https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java#L1295 .
Please note that the code above is for the trunk branch, you should go look in the branch that you are running. Thanks to great work from some folks, this code has been evolving a lot recently.

HTH
Ravi

On Tue, Aug 30, 2016 at 12:09 PM, Nico Pappagianis <[hidden email]> wrote:
How does YARN decide if remaining memory is enough AM allocation

Hi all,

Somewhat of a newb here so hopefully this question isn't too trivial.

I want to know how YARN decides if the remaining memory after AM resource allocation is "enough". Here's a breakdown of what's going on in my particular case.

We have a YARN queue that has ~1.7GB memory allocated to it. We have a parent MR job that spawns multiple child jobs. The AM container memory for the parent job is configured to 1GB. Once the AM container gets allocated its 1GB there is a remaining ~0.7GB memory available. In my case that 0.7GB is enough for the child jobs to succeed. 

My question is, what if that remaining amount wasn't 0.7GB but something much less, like 0.1GB, or 0.0001GB? How does YARN handle those cases - will it allocate any remaining resources to the child jobs? Will there be a ton of thrashing if there is only a small amount of memory available?

Thanks for any help, or a pointer in the right direction!