Configuring capacity scheduling in the face of YARN-3216

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Configuring capacity scheduling in the face of YARN-3216

Marcin Tustin
Many people will be aware of YARN-3216 (https://issues.apache.org/jira/browse/YARN-3216). In short, the capacity scheduler calculates all values only off the default label and queue.

My colleague Dave and I (and really mostly Dave!) configured our cluster to work pretty much as desired even with that bug extant. I've written up how we did it here: https://medium.com/handy-tech/practical-capacity-scheduling-with-yarn-28548ae4fb88#.5ihha0oqy

The short version is: keep the bulk of capacity under the default queue/node and get a bit clever with calculating limits. 

Marcin

Want to work at Handy? Check out our culture deck and open roles
Latest news at Handy
Handy just raised $50m led by Fidelity