Re: [DISCUSS] Merge YARN resource profile (YARN-3926) branch into trunk

Previous Topic Next Topic
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

Re: [DISCUSS] Merge YARN resource profile (YARN-3926) branch into trunk

Andrew Wang
Hi Wangda,

Can this feature be disabled? Is it on or off by default? We're 1 month
from the target release for beta1, so I don't want to introduce risk to
existing code paths. TSv2 and S3Guard and YARN Federation are all okay in
that regard.

I'm also not clear on what work is remaining, there are a lot of unresolved
subtasks still under YARN-3926.

In terms of compatibility, the design doc talks about some PB changes. Are
these stable? Is there documentation somewhere that explains what APIs are
stable or unstable for users?

I'm going to start a separate discussion about beta1 scope. I think this is
the fourth merge proposal this week, and this amount of code movement makes
me very nervous considering that beta1 is only a month out.


On Thu, Aug 17, 2017 at 8:27 PM, Wangda Tan <[hidden email]> wrote:

> +hdfs/common/mr
> On Thu, Aug 17, 2017 at 1:28 PM, Wangda Tan <[hidden email]> wrote:
> > Hi all,
> >
> > I want to hear your thoughts of merging YARN resource profile branch into
> > trunk in the next few weeks. The goal is to get it in for Hadoop 3.0
> beta1.
> >
> > *Regarding to testing:*
> > We did extensive tests for the feature in the last several months.
> > Comparing to latest trunk.
> > - For SLS benchmark: We didn't see observable performance gap from
> > simulated test based on 8K nodes SLS traces (1 PB memory). We got 3k+
> > containers allocated per second.
> > - For microbenchmark: We use performance test cases added by YARN-6775,
> it
> > shows around 5% performance regression comparing to trunk.
> >
> > *Regarding to API stability: *
> > Most new added @Public APIs are @Unstable (We're going to convert some
> new
> > added @Public/@Evolving to @Unstable in the cleanup JIRA as well), we
> want
> > to get this included by beta1 so we get some feedbacks before declaring
> > stable API.
> >
> > There're few pending cleanups under YARN-3926 umbrella JIRA. Besides
> these
> > cleanups, this feature works from end-to-end, we will do another
> iteration
> > of end-to-end tests after cleanup patches got committed.
> >
> > We would love to get your thoughts before opening a voting thread.
> >
> > Special thanks to a team of folks who worked hard and contributed towards
> > this efforts including design discussion / patch / reviews, etc.: Varun
> > Vasudev, Sunil Govind, Daniel Templeton, Vinod Vavilapalli, Yufei Gu,
> > Karthik Kambatla, Jason Lowe, Arun Suresh.
> >
> > Thanks,
> > Wangda Tan
> >