[VOTE] Merge yarn-native-services branch into trunk

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

[VOTE] Merge yarn-native-services branch into trunk

Jian He
Hi All,

I would like to call a vote for merging yarn-native-services to trunk. The vote will run for 7 days as usual.

At a high level, the following are the key feautres implemented.
- YARN-5079[1]. A native YARN framework (ApplicationMaster) to migrate and orchestrate existing services to YARN either docker or non-docker based.
- YARN-4793[2]. A Rest API server for user to deploy a service via a simple JSON spec
- YARN-4757[3]. Extending today's service registry with a simple DNS service to enable users to discover services deployed on YARN
- YARN-6419[4]. UI support for native-services on the new YARN UI
All these new services are optional and are sitting outside of the existing system, and have no impact on existing system if disabled.

Special thanks to a team of folks who worked hard towards this: Billie Rinaldi, Gour Saha, Vinod Kumar Vavilapalli, Jonathan Maron, Rohith Sharma K S, Sunil G, Akhil PB. This effort could not be possible without their ideas and hard work.

Thanks,
Jian

[1] https://issues.apache.org/jira/browse/YARN-5079
[2] https://issues.apache.org/jira/browse/YARN-4793
[3] https://issues.apache.org/jira/browse/YARN-4757
[4] https://issues.apache.org/jira/browse/YARN-6419


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Merge yarn-native-services branch into trunk

Gour Saha
+1 (non-binding)

On 9/1/17, 11:58 AM, "Billie Rinaldi" <[hidden email]> wrote:

>+1 (non-binding)
>
>On Thu, Aug 31, 2017 at 8:33 PM, Jian He <[hidden email]> wrote:
>
>> Hi All,
>>
>> I would like to call a vote for merging yarn-native-services to trunk.
>>The
>> vote will run for 7 days as usual.
>>
>> At a high level, the following are the key feautres implemented.
>> - YARN-5079[1]. A native YARN framework (ApplicationMaster) to migrate
>>and
>> orchestrate existing services to YARN either docker or non-docker based.
>> - YARN-4793[2]. A Rest API server for user to deploy a service via a
>> simple JSON spec
>> - YARN-4757[3]. Extending today's service registry with a simple DNS
>> service to enable users to discover services deployed on YARN
>> - YARN-6419[4]. UI support for native-services on the new YARN UI
>> All these new services are optional and are sitting outside of the
>> existing system, and have no impact on existing system if disabled.
>>
>> Special thanks to a team of folks who worked hard towards this: Billie
>> Rinaldi, Gour Saha, Vinod Kumar Vavilapalli, Jonathan Maron, Rohith
>>Sharma
>> K S, Sunil G, Akhil PB. This effort could not be possible without their
>> ideas and hard work.
>>
>> Thanks,
>> Jian
>>
>> [1] https://issues.apache.org/jira/browse/YARN-5079
>> [2] https://issues.apache.org/jira/browse/YARN-4793
>> [3] https://issues.apache.org/jira/browse/YARN-4757
>> [4] https://issues.apache.org/jira/browse/YARN-6419
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Merge yarn-native-services branch into trunk

Allen Wittenauer-6
In reply to this post by Jian He

> On Sep 5, 2017, at 3:12 PM, Gour Saha <[hidden email]> wrote:
>
> 2) Lots of markdown problems in the NativeServicesDiscovery.md document.
> This includes things like Œyarnsite.xml¹ (missing a dash.)
>
> The md patch uploaded to YARN-5244 had some special chars. I fixed those
> in YARN-7161.


        It’s a lot more than just special chars I think.  Even github (which has a way better markdown processor than what we’re using for the site docs) is having trouble rendering it:

https://github.com/apache/hadoop/blob/51c39c4261236ab714fe0ec8d00753dc4c6406ee/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/native-services/NativeServicesDiscovery.md

e.g., all of those ‘###’ are likely missing a space.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] Merge yarn-native-services branch into trunk

Andrew Wang
In reply to this post by Jian He
Thanks for your consideration Jian, let's track this for GA then.

Best,
Andrew

On Fri, Sep 8, 2017 at 3:02 PM, Jian He <[hidden email]> wrote:

> Hi Andrew,
>
> At this point, there are no more release blockers including documentations
> from our side - all work done.
> But I agree it is too close to the release, after talking with other team
> members, we are fine to drop  this from beta,
>
> And we want to target this for GA.
> I’m withdrawing this vote and will start afresh vote later for GA.
> Thanks all who voted this effort !
>
> Thanks,
> Jian
>
>
> > On Sep 7, 2017, at 3:59 PM, Andrew Wang <[hidden email]>
> wrote:
> >
> > Hi folks,
> >
> > This vote closes today. I see a -1 from Allen on inclusion in beta1. I
> see
> > there's active fixing going on, but given that we're one week out from
> RC0,
> > I think we should drop this from beta1.
> >
> > Allen, Jian, others, is this reasonable? What release should we retarget
> > this for? I don't have a sense for how much work there is left to do, but
> > as a reminder, we're planning GA for Nov 1st, and 3.1.0 for January.
> >
> > Best,
> > Andrew
> >
> > On Wed, Sep 6, 2017 at 10:19 AM, Jian He <[hidden email]> wrote:
> >
> >>>      Please correct me if I’m wrong, but the current summary of the
> >> branch, post these changes, looks like:
> >> Sorry for confusion, I was actively writing the formal documentation for
> >> how to use/how it works etc. and will post soon in a few hours.
> >>
> >>
> >>> On Sep 6, 2017, at 10:15 AM, Allen Wittenauer <
> [hidden email]>
> >> wrote:
> >>>
> >>>
> >>>> On Sep 5, 2017, at 6:23 PM, Jian He <[hidden email]> wrote:
> >>>>
> >>>>>    If it doesn’t have all the bells and whistles, then it shouldn’t
> >> be on port 53 by default.
> >>>> Sure, I’ll change the default port to not use 53 and document it.
> >>>>>    *how* is it getting launched on a privileged port? It sounds like
> >> the expectation is to run “command” as root.   *ALL* of the previous
> >> daemons in Hadoop that needed a privileged port used jsvc.  Why isn’t
> this
> >> one? These questions matter from a security standpoint.
> >>>> Yes, it is running as “root” to be able to use the privileged port.
> The
> >> DNS server is not yet integrated with the hadoop script.
> >>>>
> >>>>> Check the output.  It’s pretty obviously borked:
> >>>> Thanks for pointing out. Missed this when rebasing onto trunk.
> >>>
> >>>
> >>>      Please correct me if I’m wrong, but the current summary of the
> >> branch, post these changes, looks like:
> >>>
> >>>              * A bunch of mostly new Java code that may or may not have
> >> javadocs (post-revert YARN-6877, still working out HADOOP-14835)
> >>>              * ~1/3 of the docs are roadmap/TBD
> >>>              * ~1/3 of the docs are for an optional DNS daemon that has
> >> no end user hook to start it
> >>>              * ~1/3 of the docs are for a REST API that comes from some
> >> undefined daemon (apiserver?)
> >>>              * Two new, but undocumented, subcommands to yarn
> >>>              * There are no docs for admins or users on how to actually
> >> start or use this completely new/separate/optional feature
> >>>
> >>>      How are outside people (e.g., non-branch committers) supposed to
> >> test this new feature under these conditions?
> >>>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: [hidden email]
> >> For additional commands, e-mail: [hidden email]
> >>
> >>
>
>