[VOTE] - Release 2.0.5-beta

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
62 messages Options
1234
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] - Release 2.0.5-beta

Zhijie Shen-2
+1 (non-binding) on the proposal.


On Wed, May 15, 2013 at 1:43 PM, Eli Collins <[hidden email]> wrote:

> On Wed, May 15, 2013 at 1:29 PM, Matt Foley <[hidden email]>
> wrote:
>
> > >> Arun, not sure whether your "Yes to all" already covered this, but I'd
> > like
> > >> to throw in support for the compatibility guidelines being a blocker.
> >
> > +1 to that.  Definitely an overriding concern for me.
> >
> >
> +1  Likewise.   Would be great to get more eyeballs on Karthik's patch
> on HADOOP-9517 if people haven't review it already.
>



--
Zhijie Shen
Hortonworks Inc.
http://hortonworks.com/
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] - Release 2.0.5-beta

Roman Shaposhnik-2
In reply to this post by Matt Foley
On Wed, May 15, 2013 at 1:36 PM, Matt Foley <[hidden email]> wrote:
>>> lets fork this thread into the appropriate ML and discuss the practical,
> achievable
>>> steps that can be included into the release criteria of Hadoop 2.0.5-beta
>
> Seems to me common-dev is the appropriate ML,

Thanks. I'll stick to this thread then.

> and Arun has invited Jiras to include.
> Open a Jira with your suggested list, and we carry on the discussion from
> there.  Does that work?

But this is exactly my concern -- I don't have the list of JIRAs.
In fact, there's work that needs to be done to arrive at the
list of JIRAs that would be complete enough to give me
confidence that something like MAPREDUCE-5240 (I'll stick
to this example simply because I remember it by heart now ;-)).
What I'm saying is this -- if nobody is willing to do this work
outside of the very few folks who are part of Apache Bigtop
then I have very little confidence in this proposal actually
delivering on its promise of beta quality in Hadoop 2.0.5.

The question I'm asking is actually quite simple: does Hadoop
community believe in investing in doing this work to COME UP
with such a list? Or to ask it differently -- does the Hadoop
community value the feedback that such a work would provide
to a degree that it would be made part of the release criteria
for Hadoop 2.0.5-beta?

This is really a binary question as far as I can tell.

Thanks,
Roman.

P.S. There's a second level of discussion which is -- what
exactly does that extra work entail -- but lets deal with
the basic question first.
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] - Release 2.0.5-beta

Vinod Vavilapalli
In reply to this post by Roman Shaposhnik-2

Roman, I keep this same argument again and again. Should've refuted earlier.

Please list down all the issues that BigTop ran into *because of* new features. You continue to argue that new features are destabilizing 2.0.*, which I don't agree with at all. 2.0.3-alpha was the last time major features got merged in, and we found blockers irrespective of those.

MAPREDUCE-5240 specifically isn't due to any feature merge. This was a bug. I'd say this is a long standing bug in 2.0.x. You sure this passed in 2.0.3? Even so, this is mostly broken by another bug-fix and *not* because of any feature.

I quickly checked other bugs you reported in 2.0.x:
 - MAPREDUCE-5088 was caused by the fix for HADOOP-9299 which was again a long standing issue in 2.0.x
 - MAPREDUCE-3728 is similar
 - MAPREDUCE-5117 is similar
 - MAPREDUCE-4219 was a security related feature request from you.
 - MAPREDUCE-3916 was because of new proxy-server added.

I am not arguing that new features *may* destabilize the branch, but you've repeatedly stated this as if that were a fact.

Really appreciate the testing done by BigTop, but please don't distort the facts.

Thanks,
+Vinod


On May 15, 2013, at 1:29 PM, Roman Shaposhnik wrote:

> Please tell me if my expectations are incorrect, but to me the -beta would
> signify it being a 'safe' target for the downstream components. We're still
> finding *very* basic and *very* disruptive issues (MAPREDUCE-5240 is
> a good example) that essentially mean DOA for downstream that depends
> on this functionality.
>
> Are we comfortable with delivering 2.0.5-beta and later on starting
> to discover things like MAPREDUCE-5240 more or less accidentally?

Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] - Release 2.0.5-beta

Vinod Vavilapalli
Typo, keep hearing*

Thanks,
+Vinod

On May 15, 2013, at 2:14 PM, Vinod Kumar Vavilapalli wrote:

> Roman, I keep this same argument again and again. Should've refuted earlier.

Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] - Release 2.0.5-beta

Arun Murthy
In reply to this post by Vinod Vavilapalli
Great summary, thanks Vinod.

On May 15, 2013, at 2:14 PM, Vinod Kumar Vavilapalli wrote:

>
> Roman, I keep this same argument again and again. Should've refuted earlier.
>
> Please list down all the issues that BigTop ran into *because of* new features. You continue to argue that new features are destabilizing 2.0.*, which I don't agree with at all. 2.0.3-alpha was the last time major features got merged in, and we found blockers irrespective of those.
>
> MAPREDUCE-5240 specifically isn't due to any feature merge. This was a bug. I'd say this is a long standing bug in 2.0.x. You sure this passed in 2.0.3? Even so, this is mostly broken by another bug-fix and *not* because of any feature.
>
> I quickly checked other bugs you reported in 2.0.x:
> - MAPREDUCE-5088 was caused by the fix for HADOOP-9299 which was again a long standing issue in 2.0.x
> - MAPREDUCE-3728 is similar
> - MAPREDUCE-5117 is similar
> - MAPREDUCE-4219 was a security related feature request from you.
> - MAPREDUCE-3916 was because of new proxy-server added.
>
> I am not arguing that new features *may* destabilize the branch, but you've repeatedly stated this as if that were a fact.
>
> Really appreciate the testing done by BigTop, but please don't distort the facts.
>
> Thanks,
> +Vinod
>
>
> On May 15, 2013, at 1:29 PM, Roman Shaposhnik wrote:
>
>> Please tell me if my expectations are incorrect, but to me the -beta would
>> signify it being a 'safe' target for the downstream components. We're still
>> finding *very* basic and *very* disruptive issues (MAPREDUCE-5240 is
>> a good example) that essentially mean DOA for downstream that depends
>> on this functionality.
>>
>> Are we comfortable with delivering 2.0.5-beta and later on starting
>> to discover things like MAPREDUCE-5240 more or less accidentally?
>

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/


Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] - Release 2.0.5-beta

Steve Loughran-3
In reply to this post by Arun Murthy
On 15 May 2013 10:57, Arun C Murthy <[hidden email]> wrote:

> Folks,
>
> A considerable number of people have expressed confusion regarding the
> recent vote on 2.0.5, beta status etc. given lack of specifics, the voting
> itself (validity of the vote itself, whose votes are binding) etc.
>
> IMHO technical arguments (incompatibility b/w 2.0 & 2.1, current stability
> of 3 features under debate etc.) have been lost in the discussion in favor
> of non-technical (almost dramatic) nuances such as "seizing the moment".
> There is now dangerous talk of tolerating incompatibility b/w 2.0 and 2.1)
> - this is a red flag for me; particularly when there are just 3 features
> being debated and active committers and contributors are confident of and
> ready to stand by their work. All patches, I believe, are ready to be
> merged in the the next few days per discussions on jira. This will,
> clearly, not delay the other API work which everyone agrees is crucial. As
> a result, I feel no recourse but to restart a new vote - all attempts at
> calm, reasoned, civil discussion based on technical arguments have come to
> naught - I apologize for the thrash caused to everyone's attention.
>
> To get past all of this confusion, I'd like to present an alternate,
> specific proposal for consideration.
>
> I propose we continue the original plan and make a 2.0.5-beta release by
> May end with the following content:
> # HDFS-347
> # HDFS Snapshots
> # Windows support
> # Necessary & final API/protocol changes such as:
>  * Final YARN API changes: YARN-386
>  * MR Binary Compatibility: MAPREDUCE-5108
>  * Final RPC cleanup: HADOOP-8990
>
> People working on the above features have all expressed considerable
> comfort with them and are ready to stand-by to help expedite any necessary
> bug-fixes etc. to get to stabilization quickly. I'm confident we can get
> this release out by end of May. This sets stage for a hadoop-2.x GA release
> right after with some more testing - this means I think I can quickly turn
> around and make bug-fix releases as necessary right after 2.0.5-beta.
>
> I request that people consider helping out with this plan and sign up to
> help push hadoop-2.x to stability as outlined above. I believe this will
> help achieve our shared goals of quickly stabilizing hadoop-2 and help
> ensure we can support it for forseeable future in a compatible manner for
> the benefit of our users and downstream projects.
>
> Please vote, the vote will run the normal 7 days. Obviously, I'm +1.
>

+1 (binding)
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] - Release 2.0.5-beta

Roman Shaposhnik-2
In reply to this post by Vinod Vavilapalli
On Wed, May 15, 2013 at 2:14 PM, Vinod Kumar Vavilapalli
<[hidden email]> wrote:
> Please list down all the issues that BigTop ran into *because of* new features.

Whether the bug is *because of* new feature or not is a red herring
for my argument. Please lets drop this distinction. I never used it.

> You continue to argue that new features are destabilizing 2.0.*,
> which I don't agree with at all. 2.0.3-alpha was the last time major
> features got merged in, and we found blockers irrespective of those.

This is not my argument at all. I apologize if somehow I failed to
communicate it, but here's what my argument boils down to:
given *my* experience with Hadoop 2.0.x series and Bigtop
release every time I try a different release of Hadoop 2.0.x
I run into issues that scare me. They scare me because
they are so basic yet they make component like Sqoop
and Oozie (and I believe Giraph on one occasion)
pretty much DOA for YARN-base mapreduce implementation.

In my mind, what that translates into is the fact that nobody
did *any* real testing of a particular downstream component
running on a given Hadoop 2.0.x release. Like I said --
the issues so far make the components in question DOA.

Effectively the onion of issues remain unpeeled, so to speak.

What I'm asking on this thread (and somehow nobody is willing
to give me a straight answer) is whether the Hadoop community
is willing to invest in peeling this onion of issues somewhat more
before declaring Hadoop 2.0.5 a beta release.

Once again it is a binary question -- please give me an answer
of yes or no.

> I am not arguing that new features *may* destabilize the branch, but you've repeatedly stated this as if that were a fact.

Your list of issues is pretty complete (give or take a few that I didn't file
but Cos and others did). And I'd be the first one to agree that
it is not a large list of issues. What scares me is not its size,
but the fact how basic they are and how the block the *rest*
of the testing completely.

To be extra clear -- what scares me about something like
MAPREDUCE-5240 is not whether it came as a result of
a merge or was sitting there since day one. What scares
me is that we've identified it last week and yet Sqoop 2 is
DOA in its presense.

How many more issues like that one (regardless of how
they originated) are in branch-2? Wouldn't we want to
know before declaring Hadoop 2.0.5 beta?

Now, knowing would require work -- that's what my
argument is all about.


Thanks,
Roman.
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] - Release 2.0.5-beta

Arun Murthy
In reply to this post by Arun Murthy
Roman,

Furthermore, before we rush into finding flaws and scaring kids at night it would be useful to remember one thing:
Software has *bugs*. We can't block any release till the entire universe validates it, in fact they won't validate it if we don't release since are at the bottom of the stack.

Any help prior to the release is welcome; I know people who work for the same employer as I do have plans to do further testing after we freeze apis via the beta release(s). I hope and pray others can join this effort - thanks to everyone who already has.

Again, freezing APIs and protocols is the primary aim of 2.0.5-beta. There are no guarantees it's 100% bug-free, we can never make such guarantees anyway.

If, and when, we find bugs with 2.0.5-beta I'm more than happy to quickly turn around and make more releases (2.0.6-beta, 2.0.7-beta). Obviously I'll make a call on which bugs are critical - feedback to help me decide is, as always, welcome.
I've been clear, many times, that we might need more than one beta release to iron out bugs etc.

None of this should be a surprise - this has happened many, many times in the lifetime of this and other projects. 2.0.3-alpha vis-a-vis 2.0.4-alpha is the most recent example - it won't be the last.

So, I hope, concludes this meme.

thanks,
Arun

On May 15, 2013, at 2:20 PM, Arun C Murthy wrote:

> Great summary, thanks Vinod.
>
> On May 15, 2013, at 2:14 PM, Vinod Kumar Vavilapalli wrote:
>
>>
>> Roman, I keep this same argument again and again. Should've refuted earlier.
>>
>> Please list down all the issues that BigTop ran into *because of* new features. You continue to argue that new features are destabilizing 2.0.*, which I don't agree with at all. 2.0.3-alpha was the last time major features got merged in, and we found blockers irrespective of those.
>>
>> MAPREDUCE-5240 specifically isn't due to any feature merge. This was a bug. I'd say this is a long standing bug in 2.0.x. You sure this passed in 2.0.3? Even so, this is mostly broken by another bug-fix and *not* because of any feature.
>>
>> I quickly checked other bugs you reported in 2.0.x:
>> - MAPREDUCE-5088 was caused by the fix for HADOOP-9299 which was again a long standing issue in 2.0.x
>> - MAPREDUCE-3728 is similar
>> - MAPREDUCE-5117 is similar
>> - MAPREDUCE-4219 was a security related feature request from you.
>> - MAPREDUCE-3916 was because of new proxy-server added.
>>
>> I am not arguing that new features *may* destabilize the branch, but you've repeatedly stated this as if that were a fact.
>>
>> Really appreciate the testing done by BigTop, but please don't distort the facts.
>>
>> Thanks,
>> +Vinod
>>
>>
>> On May 15, 2013, at 1:29 PM, Roman Shaposhnik wrote:
>>
>>> Please tell me if my expectations are incorrect, but to me the -beta would
>>> signify it being a 'safe' target for the downstream components. We're still
>>> finding *very* basic and *very* disruptive issues (MAPREDUCE-5240 is
>>> a good example) that essentially mean DOA for downstream that depends
>>> on this functionality.
>>>
>>> Are we comfortable with delivering 2.0.5-beta and later on starting
>>> to discover things like MAPREDUCE-5240 more or less accidentally?
>>
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/


Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] - Release 2.0.5-beta

Chris Douglas
In reply to this post by Steve Loughran-3
+1 (binding) on the proposal.

However, the value we get from these "release plan" votes is dubious,
to put it mildly. The surrounding discussion has cost more than it is
worth, and votes on executive summaries of releases discourage the
sort of detailed collaboration we're trying to create. It replaces
development with zero-sum struggles over abstractions.

This is, in effect, another poll about the direction we're taking 2.x.
If we can't reach consensus on development directions without voting,
that's more evidence that the project should be split, IMO. -C

On Wed, May 15, 2013 at 2:21 PM, Steve Loughran <[hidden email]> wrote:

> On 15 May 2013 10:57, Arun C Murthy <[hidden email]> wrote:
>
>> Folks,
>>
>> A considerable number of people have expressed confusion regarding the
>> recent vote on 2.0.5, beta status etc. given lack of specifics, the voting
>> itself (validity of the vote itself, whose votes are binding) etc.
>>
>> IMHO technical arguments (incompatibility b/w 2.0 & 2.1, current stability
>> of 3 features under debate etc.) have been lost in the discussion in favor
>> of non-technical (almost dramatic) nuances such as "seizing the moment".
>> There is now dangerous talk of tolerating incompatibility b/w 2.0 and 2.1)
>> - this is a red flag for me; particularly when there are just 3 features
>> being debated and active committers and contributors are confident of and
>> ready to stand by their work. All patches, I believe, are ready to be
>> merged in the the next few days per discussions on jira. This will,
>> clearly, not delay the other API work which everyone agrees is crucial. As
>> a result, I feel no recourse but to restart a new vote - all attempts at
>> calm, reasoned, civil discussion based on technical arguments have come to
>> naught - I apologize for the thrash caused to everyone's attention.
>>
>> To get past all of this confusion, I'd like to present an alternate,
>> specific proposal for consideration.
>>
>> I propose we continue the original plan and make a 2.0.5-beta release by
>> May end with the following content:
>> # HDFS-347
>> # HDFS Snapshots
>> # Windows support
>> # Necessary & final API/protocol changes such as:
>>  * Final YARN API changes: YARN-386
>>  * MR Binary Compatibility: MAPREDUCE-5108
>>  * Final RPC cleanup: HADOOP-8990
>>
>> People working on the above features have all expressed considerable
>> comfort with them and are ready to stand-by to help expedite any necessary
>> bug-fixes etc. to get to stabilization quickly. I'm confident we can get
>> this release out by end of May. This sets stage for a hadoop-2.x GA release
>> right after with some more testing - this means I think I can quickly turn
>> around and make bug-fix releases as necessary right after 2.0.5-beta.
>>
>> I request that people consider helping out with this plan and sign up to
>> help push hadoop-2.x to stability as outlined above. I believe this will
>> help achieve our shared goals of quickly stabilizing hadoop-2 and help
>> ensure we can support it for forseeable future in a compatible manner for
>> the benefit of our users and downstream projects.
>>
>> Please vote, the vote will run the normal 7 days. Obviously, I'm +1.
>>
>
> +1 (binding)
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] - Release 2.0.5-beta

Steve Loughran-3
In reply to this post by Arun Murthy
On 15 May 2013 15:02, Arun C Murthy <[hidden email]> wrote:

> Roman,
>
> Furthermore, before we rush into finding flaws and scaring kids at night
> it would be useful to remember one thing:
> Software has *bugs*. We can't block any release till the entire universe
> validates it, in fact they won't validate it if we don't release since are
> at the bottom of the stack.
>
>

more subtly: we aren't going to find all the corner case situations until
things ship into the hands of people whose {networks, configs,
applications, hardware} are different. Marking something as -beta means
more people will use it, and find those problems, at a time when it is
still possible for a  fast turnaround on fixes. what we are implicitly
saying with a "-beta" tag is " ready for others to use", which in Hadoop's
case means "doesn't lose data unless you do something suicidal" and "we're
not going to move APIs on you". The gulf from -beta to shipping is usually
much less dramatic than -alpha to -beta, as it happens when everyone is
happy that the last beta is good enough to push out.

-Steve

(who will be at the HUG in Sunnyvale this evening)
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] - Release 2.0.5-beta

Konstantin Shvachko-2
In reply to this post by Arun Murthy
Arun,

I am glad I at least convinced you to finally announce your release plan
and put it into vote.
Even though it is to overrule the vote that just completed, which you were
against and lost, well - Twice.

I am glad you removed the NFS feature from the list proposed earlier.

I think this vote is late. The lazy consensus on that issue has been just
reached.
I don't see the basis for the new vote,
and it is not clear what action you seek to approve.

Thanks,
--Konstantin



On Wed, May 15, 2013 at 10:57 AM, Arun C Murthy <[hidden email]> wrote:

> Folks,
>
> A considerable number of people have expressed confusion regarding the
> recent vote on 2.0.5, beta status etc. given lack of specifics, the voting
> itself (validity of the vote itself, whose votes are binding) etc.
>
> IMHO technical arguments (incompatibility b/w 2.0 & 2.1, current stability
> of 3 features under debate etc.) have been lost in the discussion in favor
> of non-technical (almost dramatic) nuances such as "seizing the moment".
> There is now dangerous talk of tolerating incompatibility b/w 2.0 and 2.1)
> - this is a red flag for me; particularly when there are just 3 features
> being debated and active committers and contributors are confident of and
> ready to stand by their work. All patches, I believe, are ready to be
> merged in the the next few days per discussions on jira. This will,
> clearly, not delay the other API work which everyone agrees is crucial. As
> a result, I feel no recourse but to restart a new vote - all attempts at
> calm, reasoned, civil discussion based on technical arguments have come to
> naught - I apologize for the thrash caused to everyone's attention.
>
> To get past all of this confusion, I'd like to present an alternate,
> specific proposal for consideration.
>
> I propose we continue the original plan and make a 2.0.5-beta release by
> May end with the following content:
> # HDFS-347
> # HDFS Snapshots
> # Windows support
> # Necessary & final API/protocol changes such as:
>  * Final YARN API changes: YARN-386
>  * MR Binary Compatibility: MAPREDUCE-5108
>  * Final RPC cleanup: HADOOP-8990
>
> People working on the above features have all expressed considerable
> comfort with them and are ready to stand-by to help expedite any necessary
> bug-fixes etc. to get to stabilization quickly. I'm confident we can get
> this release out by end of May. This sets stage for a hadoop-2.x GA release
> right after with some more testing - this means I think I can quickly turn
> around and make bug-fix releases as necessary right after 2.0.5-beta.
>
> I request that people consider helping out with this plan and sign up to
> help push hadoop-2.x to stability as outlined above. I believe this will
> help achieve our shared goals of quickly stabilizing hadoop-2 and help
> ensure we can support it for forseeable future in a compatible manner for
> the benefit of our users and downstream projects.
>
> Please vote, the vote will run the normal 7 days. Obviously, I'm +1.
>
> thanks,
> Arun
>
> PS: To keep this discussion grounded in technical details I've moved this
> to dev@ (bcc general@).
>
>
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] - Release 2.0.5-beta

Roman Shaposhnik-2
In reply to this post by Arun Murthy
Arun,

am I reading yours answer to my binary question correctly? It is a 'no'.

My reading of your response is that while you appreciate the feedback
Bigtop is providing you're not of an opinion that investigating the level
of stability of Hadoop wrt. downstream any further than what is currently
happening would be a worthy investment of Hadoop's community
(or your personal for that matter) time?

Thanks,
Roman.

On Wed, May 15, 2013 at 3:02 PM, Arun C Murthy <[hidden email]> wrote:

> Roman,
>
> Furthermore, before we rush into finding flaws and scaring kids at night it would be useful to remember one thing:
> Software has *bugs*. We can't block any release till the entire universe validates it, in fact they won't validate it if we don't release since are at the bottom of the stack.
>
> Any help prior to the release is welcome; I know people who work for the same employer as I do have plans to do further testing after we freeze apis via the beta release(s). I hope and pray others can join this effort - thanks to everyone who already has.
>
> Again, freezing APIs and protocols is the primary aim of 2.0.5-beta. There are no guarantees it's 100% bug-free, we can never make such guarantees anyway.
>
> If, and when, we find bugs with 2.0.5-beta I'm more than happy to quickly turn around and make more releases (2.0.6-beta, 2.0.7-beta). Obviously I'll make a call on which bugs are critical - feedback to help me decide is, as always, welcome.
> I've been clear, many times, that we might need more than one beta release to iron out bugs etc.
>
> None of this should be a surprise - this has happened many, many times in the lifetime of this and other projects. 2.0.3-alpha vis-a-vis 2.0.4-alpha is the most recent example - it won't be the last.
>
> So, I hope, concludes this meme.
>
> thanks,
> Arun
>
> On May 15, 2013, at 2:20 PM, Arun C Murthy wrote:
>
>> Great summary, thanks Vinod.
>>
>> On May 15, 2013, at 2:14 PM, Vinod Kumar Vavilapalli wrote:
>>
>>>
>>> Roman, I keep this same argument again and again. Should've refuted earlier.
>>>
>>> Please list down all the issues that BigTop ran into *because of* new features. You continue to argue that new features are destabilizing 2.0.*, which I don't agree with at all. 2.0.3-alpha was the last time major features got merged in, and we found blockers irrespective of those.
>>>
>>> MAPREDUCE-5240 specifically isn't due to any feature merge. This was a bug. I'd say this is a long standing bug in 2.0.x. You sure this passed in 2.0.3? Even so, this is mostly broken by another bug-fix and *not* because of any feature.
>>>
>>> I quickly checked other bugs you reported in 2.0.x:
>>> - MAPREDUCE-5088 was caused by the fix for HADOOP-9299 which was again a long standing issue in 2.0.x
>>> - MAPREDUCE-3728 is similar
>>> - MAPREDUCE-5117 is similar
>>> - MAPREDUCE-4219 was a security related feature request from you.
>>> - MAPREDUCE-3916 was because of new proxy-server added.
>>>
>>> I am not arguing that new features *may* destabilize the branch, but you've repeatedly stated this as if that were a fact.
>>>
>>> Really appreciate the testing done by BigTop, but please don't distort the facts.
>>>
>>> Thanks,
>>> +Vinod
>>>
>>>
>>> On May 15, 2013, at 1:29 PM, Roman Shaposhnik wrote:
>>>
>>>> Please tell me if my expectations are incorrect, but to me the -beta would
>>>> signify it being a 'safe' target for the downstream components. We're still
>>>> finding *very* basic and *very* disruptive issues (MAPREDUCE-5240 is
>>>> a good example) that essentially mean DOA for downstream that depends
>>>> on this functionality.
>>>>
>>>> Are we comfortable with delivering 2.0.5-beta and later on starting
>>>> to discover things like MAPREDUCE-5240 more or less accidentally?
>>>
>>
>> --
>> Arun C. Murthy
>> Hortonworks Inc.
>> http://hortonworks.com/
>>
>>
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] - Release 2.0.5-beta

Konstantin Boudnik-2
In reply to this post by Roman Shaposhnik-2
Indeed. I think the root of the issue is deeper. ASF software practices are
great to deal with isolated, relatively contained projects like httpd,
libreoffice, trac, etc. However, Hadoop based stack - essentially, software
aimed at enterprises with bigger scale operations - is a different animal,
that requires balancing of a huge number of moving parts and an unbroken
flow of feedback up the stream. Anyone who have delivered any enterprise
grade software system knows perfectly well how hard is that.

However, in the environment where a release pushed out in the rush
(essentially causing DOA issues downstream), these are got fixed in consequent
releases.  That ironically is likely to contain some other DOAs because an
integration testing - and I mean real world integration system testing - is
done by this small project, that is treated like a toy for adolescent kids.
And there's no other real integration testing happening OPENLY on the full
stack. Despite numerous claims, that is.

Software comes with bugs - this is a somewhat expected phenomena. However, bug
fixes shouldn't be mixed with new features, increasing entropy in the system.
In other words, the development process should fan-in. A process with multiple
consequent stable releases helps to achieve it; and compatibility issues would
be addressed by working on the next major release.

The model above leaves downstream with a choice of sticking to the 3.x or switching
to 4.x and so on. Where's having permanent alpha tag is a convenient way to control
software project that effectively became a vendor-controlled effort.

And yes - this leads to fragmentation, makes no mistakes about it. Because no
one can sit on the hands for a year and wait until a usable release with all
great features will come about: lot of organizations just silently forking away
to make their own environment suitable for production or sale; some of them
might sporadically contribute something back. And of course - this is not the
aim of Apache project to produce commercial grade platform.

Cos

On Wed, May 15, 2013 at 02:54PM, Roman Shaposhnik wrote:

> On Wed, May 15, 2013 at 2:14 PM, Vinod Kumar Vavilapalli
> <[hidden email]> wrote:
> > Please list down all the issues that BigTop ran into *because of* new features.
>
> Whether the bug is *because of* new feature or not is a red herring
> for my argument. Please lets drop this distinction. I never used it.
>
> > You continue to argue that new features are destabilizing 2.0.*,
> > which I don't agree with at all. 2.0.3-alpha was the last time major
> > features got merged in, and we found blockers irrespective of those.
>
> This is not my argument at all. I apologize if somehow I failed to
> communicate it, but here's what my argument boils down to:
> given *my* experience with Hadoop 2.0.x series and Bigtop
> release every time I try a different release of Hadoop 2.0.x
> I run into issues that scare me. They scare me because
> they are so basic yet they make component like Sqoop
> and Oozie (and I believe Giraph on one occasion)
> pretty much DOA for YARN-base mapreduce implementation.
>
> In my mind, what that translates into is the fact that nobody
> did *any* real testing of a particular downstream component
> running on a given Hadoop 2.0.x release. Like I said --
> the issues so far make the components in question DOA.

> Effectively the onion of issues remain unpeeled, so to speak.
>
> What I'm asking on this thread (and somehow nobody is willing
> to give me a straight answer) is whether the Hadoop community
> is willing to invest in peeling this onion of issues somewhat more
> before declaring Hadoop 2.0.5 a beta release.
>
> Once again it is a binary question -- please give me an answer
> of yes or no.
>
> > I am not arguing that new features *may* destabilize the branch, but you've repeatedly stated this as if that were a fact.
>
> Your list of issues is pretty complete (give or take a few that I didn't file
> but Cos and others did). And I'd be the first one to agree that
> it is not a large list of issues. What scares me is not its size,
> but the fact how basic they are and how the block the *rest*
> of the testing completely.
>
> To be extra clear -- what scares me about something like
> MAPREDUCE-5240 is not whether it came as a result of
> a merge or was sitting there since day one. What scares
> me is that we've identified it last week and yet Sqoop 2 is
> DOA in its presense.
>
> How many more issues like that one (regardless of how
> they originated) are in branch-2? Wouldn't we want to
> know before declaring Hadoop 2.0.5 beta?
>
> Now, knowing would require work -- that's what my
> argument is all about.
>
>
> Thanks,
> Roman.
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] - Release 2.0.5-beta

Arun Murthy
In reply to this post by Chris Douglas

On May 15, 2013, at 3:27 PM, Chris Douglas wrote:

> +1 (binding) on the proposal.
>
> However, the value we get from these "release plan" votes is dubious,
> to put it mildly. The surrounding discussion has cost more than it is
> worth, and votes on executive summaries of releases discourage the
> sort of detailed collaboration we're trying to create. It replaces
> development with zero-sum struggles over abstractions.

Agree, I propose we edit bylaws to do away with them for the future.
>
> This is, in effect, another poll about the direction we're taking 2.x.
> If we can't reach consensus on development directions without voting,
> that's more evidence that the project should be split, IMO. -C

+1e100

Arun

>
> On Wed, May 15, 2013 at 2:21 PM, Steve Loughran <[hidden email]> wrote:
>> On 15 May 2013 10:57, Arun C Murthy <[hidden email]> wrote:
>>
>>> Folks,
>>>
>>> A considerable number of people have expressed confusion regarding the
>>> recent vote on 2.0.5, beta status etc. given lack of specifics, the voting
>>> itself (validity of the vote itself, whose votes are binding) etc.
>>>
>>> IMHO technical arguments (incompatibility b/w 2.0 & 2.1, current stability
>>> of 3 features under debate etc.) have been lost in the discussion in favor
>>> of non-technical (almost dramatic) nuances such as "seizing the moment".
>>> There is now dangerous talk of tolerating incompatibility b/w 2.0 and 2.1)
>>> - this is a red flag for me; particularly when there are just 3 features
>>> being debated and active committers and contributors are confident of and
>>> ready to stand by their work. All patches, I believe, are ready to be
>>> merged in the the next few days per discussions on jira. This will,
>>> clearly, not delay the other API work which everyone agrees is crucial. As
>>> a result, I feel no recourse but to restart a new vote - all attempts at
>>> calm, reasoned, civil discussion based on technical arguments have come to
>>> naught - I apologize for the thrash caused to everyone's attention.
>>>
>>> To get past all of this confusion, I'd like to present an alternate,
>>> specific proposal for consideration.
>>>
>>> I propose we continue the original plan and make a 2.0.5-beta release by
>>> May end with the following content:
>>> # HDFS-347
>>> # HDFS Snapshots
>>> # Windows support
>>> # Necessary & final API/protocol changes such as:
>>> * Final YARN API changes: YARN-386
>>> * MR Binary Compatibility: MAPREDUCE-5108
>>> * Final RPC cleanup: HADOOP-8990
>>>
>>> People working on the above features have all expressed considerable
>>> comfort with them and are ready to stand-by to help expedite any necessary
>>> bug-fixes etc. to get to stabilization quickly. I'm confident we can get
>>> this release out by end of May. This sets stage for a hadoop-2.x GA release
>>> right after with some more testing - this means I think I can quickly turn
>>> around and make bug-fix releases as necessary right after 2.0.5-beta.
>>>
>>> I request that people consider helping out with this plan and sign up to
>>> help push hadoop-2.x to stability as outlined above. I believe this will
>>> help achieve our shared goals of quickly stabilizing hadoop-2 and help
>>> ensure we can support it for forseeable future in a compatible manner for
>>> the benefit of our users and downstream projects.
>>>
>>> Please vote, the vote will run the normal 7 days. Obviously, I'm +1.
>>>
>>
>> +1 (binding)

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/


Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] - Release 2.0.5-beta

Arun Murthy
In reply to this post by Roman Shaposhnik-2

On May 15, 2013, at 3:50 PM, Roman Shaposhnik wrote:

> Arun,
>
> am I reading yours answer to my binary question correctly? It is a 'no'.

No.

>
> My reading of your response is that while you appreciate the feedback
> Bigtop is providing you're not of an opinion that investigating the level
> of stability of Hadoop wrt. downstream any further than what is currently
> happening would be a worthy investment of Hadoop's community
> (or your personal for that matter) time?

Everyone is welcome to contribute in any and all manner. I can't speak for everyone.
It would be useful if Bigtop could run regressions on releases here consistently. We've also talked in the past about running Bigtop on branch-2, nightly. Is that something you could help with? You'd earn my personal gratitude.

thanks,
Arun

>
> Thanks,
> Roman.
>
> On Wed, May 15, 2013 at 3:02 PM, Arun C Murthy <[hidden email]> wrote:
>> Roman,
>>
>> Furthermore, before we rush into finding flaws and scaring kids at night it would be useful to remember one thing:
>> Software has *bugs*. We can't block any release till the entire universe validates it, in fact they won't validate it if we don't release since are at the bottom of the stack.
>>
>> Any help prior to the release is welcome; I know people who work for the same employer as I do have plans to do further testing after we freeze apis via the beta release(s). I hope and pray others can join this effort - thanks to everyone who already has.
>>
>> Again, freezing APIs and protocols is the primary aim of 2.0.5-beta. There are no guarantees it's 100% bug-free, we can never make such guarantees anyway.
>>
>> If, and when, we find bugs with 2.0.5-beta I'm more than happy to quickly turn around and make more releases (2.0.6-beta, 2.0.7-beta). Obviously I'll make a call on which bugs are critical - feedback to help me decide is, as always, welcome.
>> I've been clear, many times, that we might need more than one beta release to iron out bugs etc.
>>
>> None of this should be a surprise - this has happened many, many times in the lifetime of this and other projects. 2.0.3-alpha vis-a-vis 2.0.4-alpha is the most recent example - it won't be the last.
>>
>> So, I hope, concludes this meme.
>>
>> thanks,
>> Arun
>>
>> On May 15, 2013, at 2:20 PM, Arun C Murthy wrote:
>>
>>> Great summary, thanks Vinod.
>>>
>>> On May 15, 2013, at 2:14 PM, Vinod Kumar Vavilapalli wrote:
>>>
>>>>
>>>> Roman, I keep this same argument again and again. Should've refuted earlier.
>>>>
>>>> Please list down all the issues that BigTop ran into *because of* new features. You continue to argue that new features are destabilizing 2.0.*, which I don't agree with at all. 2.0.3-alpha was the last time major features got merged in, and we found blockers irrespective of those.
>>>>
>>>> MAPREDUCE-5240 specifically isn't due to any feature merge. This was a bug. I'd say this is a long standing bug in 2.0.x. You sure this passed in 2.0.3? Even so, this is mostly broken by another bug-fix and *not* because of any feature.
>>>>
>>>> I quickly checked other bugs you reported in 2.0.x:
>>>> - MAPREDUCE-5088 was caused by the fix for HADOOP-9299 which was again a long standing issue in 2.0.x
>>>> - MAPREDUCE-3728 is similar
>>>> - MAPREDUCE-5117 is similar
>>>> - MAPREDUCE-4219 was a security related feature request from you.
>>>> - MAPREDUCE-3916 was because of new proxy-server added.
>>>>
>>>> I am not arguing that new features *may* destabilize the branch, but you've repeatedly stated this as if that were a fact.
>>>>
>>>> Really appreciate the testing done by BigTop, but please don't distort the facts.
>>>>
>>>> Thanks,
>>>> +Vinod
>>>>
>>>>
>>>> On May 15, 2013, at 1:29 PM, Roman Shaposhnik wrote:
>>>>
>>>>> Please tell me if my expectations are incorrect, but to me the -beta would
>>>>> signify it being a 'safe' target for the downstream components. We're still
>>>>> finding *very* basic and *very* disruptive issues (MAPREDUCE-5240 is
>>>>> a good example) that essentially mean DOA for downstream that depends
>>>>> on this functionality.
>>>>>
>>>>> Are we comfortable with delivering 2.0.5-beta and later on starting
>>>>> to discover things like MAPREDUCE-5240 more or less accidentally?
>>>>
>>>
>>> --
>>> Arun C. Murthy
>>> Hortonworks Inc.
>>> http://hortonworks.com/
>>>
>>>
>>
>> --
>> Arun C. Murthy
>> Hortonworks Inc.
>> http://hortonworks.com/
>>
>>

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/


Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] - Release 2.0.5-beta

Matt Foley-2
Roman, what is your model for how test results from Bigtop should feed back
into Hadoop-2 development?
With the understanding that (a) software does have bugs, and (b) you're not
going to get an SLA on community-sponsored software,
what are your ideas for how to close the loop better?

Would "CI" runs of Bigtop against branch-2 be feasible, as Arun suggests?
How should we accomodate changes in individual components (Hadoop Core, but
others as well) that may require changes in one or more other components?
How does Bigtop keep doing a viable nightly build in that chaotic
environment?
Is this a previously solved problem?

Thanks,
--Matt


On Wed, May 15, 2013 at 4:47 PM, Arun C Murthy <[hidden email]> wrote:

>
> On May 15, 2013, at 3:50 PM, Roman Shaposhnik wrote:
>
> > Arun,
> >
> > am I reading yours answer to my binary question correctly? It is a 'no'.
>
> No.
>
> >
> > My reading of your response is that while you appreciate the feedback
> > Bigtop is providing you're not of an opinion that investigating the level
> > of stability of Hadoop wrt. downstream any further than what is currently
> > happening would be a worthy investment of Hadoop's community
> > (or your personal for that matter) time?
>
> Everyone is welcome to contribute in any and all manner. I can't speak for
> everyone.
> It would be useful if Bigtop could run regressions on releases here
> consistently. We've also talked in the past about running Bigtop on
> branch-2, nightly. Is that something you could help with? You'd earn my
> personal gratitude.
>
> thanks,
> Arun
>
> >
> > Thanks,
> > Roman.
> >
> > On Wed, May 15, 2013 at 3:02 PM, Arun C Murthy <[hidden email]>
> wrote:
> >> Roman,
> >>
> >> Furthermore, before we rush into finding flaws and scaring kids at
> night it would be useful to remember one thing:
> >> Software has *bugs*. We can't block any release till the entire
> universe validates it, in fact they won't validate it if we don't release
> since are at the bottom of the stack.
> >>
> >> Any help prior to the release is welcome; I know people who work for
> the same employer as I do have plans to do further testing after we freeze
> apis via the beta release(s). I hope and pray others can join this effort -
> thanks to everyone who already has.
> >>
> >> Again, freezing APIs and protocols is the primary aim of 2.0.5-beta.
> There are no guarantees it's 100% bug-free, we can never make such
> guarantees anyway.
> >>
> >> If, and when, we find bugs with 2.0.5-beta I'm more than happy to
> quickly turn around and make more releases (2.0.6-beta, 2.0.7-beta).
> Obviously I'll make a call on which bugs are critical - feedback to help me
> decide is, as always, welcome.
> >> I've been clear, many times, that we might need more than one beta
> release to iron out bugs etc.
> >>
> >> None of this should be a surprise - this has happened many, many times
> in the lifetime of this and other projects. 2.0.3-alpha vis-a-vis
> 2.0.4-alpha is the most recent example - it won't be the last.
> >>
> >> So, I hope, concludes this meme.
> >>
> >> thanks,
> >> Arun
> >>
> >> On May 15, 2013, at 2:20 PM, Arun C Murthy wrote:
> >>
> >>> Great summary, thanks Vinod.
> >>>
> >>> On May 15, 2013, at 2:14 PM, Vinod Kumar Vavilapalli wrote:
> >>>
> >>>>
> >>>> Roman, I keep this same argument again and again. Should've refuted
> earlier.
> >>>>
> >>>> Please list down all the issues that BigTop ran into *because of* new
> features. You continue to argue that new features are destabilizing 2.0.*,
> which I don't agree with at all. 2.0.3-alpha was the last time major
> features got merged in, and we found blockers irrespective of those.
> >>>>
> >>>> MAPREDUCE-5240 specifically isn't due to any feature merge. This was
> a bug. I'd say this is a long standing bug in 2.0.x. You sure this passed
> in 2.0.3? Even so, this is mostly broken by another bug-fix and *not*
> because of any feature.
> >>>>
> >>>> I quickly checked other bugs you reported in 2.0.x:
> >>>> - MAPREDUCE-5088 was caused by the fix for HADOOP-9299 which was
> again a long standing issue in 2.0.x
> >>>> - MAPREDUCE-3728 is similar
> >>>> - MAPREDUCE-5117 is similar
> >>>> - MAPREDUCE-4219 was a security related feature request from you.
> >>>> - MAPREDUCE-3916 was because of new proxy-server added.
> >>>>
> >>>> I am not arguing that new features *may* destabilize the branch, but
> you've repeatedly stated this as if that were a fact.
> >>>>
> >>>> Really appreciate the testing done by BigTop, but please don't
> distort the facts.
> >>>>
> >>>> Thanks,
> >>>> +Vinod
> >>>>
> >>>>
> >>>> On May 15, 2013, at 1:29 PM, Roman Shaposhnik wrote:
> >>>>
> >>>>> Please tell me if my expectations are incorrect, but to me the -beta
> would
> >>>>> signify it being a 'safe' target for the downstream components.
> We're still
> >>>>> finding *very* basic and *very* disruptive issues (MAPREDUCE-5240 is
> >>>>> a good example) that essentially mean DOA for downstream that depends
> >>>>> on this functionality.
> >>>>>
> >>>>> Are we comfortable with delivering 2.0.5-beta and later on starting
> >>>>> to discover things like MAPREDUCE-5240 more or less accidentally?
> >>>>
> >>>
> >>> --
> >>> Arun C. Murthy
> >>> Hortonworks Inc.
> >>> http://hortonworks.com/
> >>>
> >>>
> >>
> >> --
> >> Arun C. Murthy
> >> Hortonworks Inc.
> >> http://hortonworks.com/
> >>
> >>
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] - Release 2.0.5-beta

Matt Foley-2
In reply to this post by Arun Murthy
I'm actually drafting such a proposal.  Will open the discussion as a
[PROPOSAL] in general@
--Matt


On Wed, May 15, 2013 at 4:44 PM, Arun C Murthy <[hidden email]> wrote:

>
> On May 15, 2013, at 3:27 PM, Chris Douglas wrote:
>
> > +1 (binding) on the proposal.
> >
> > However, the value we get from these "release plan" votes is dubious,
> > to put it mildly. The surrounding discussion has cost more than it is
> > worth, and votes on executive summaries of releases discourage the
> > sort of detailed collaboration we're trying to create. It replaces
> > development with zero-sum struggles over abstractions.
>
> Agree, I propose we edit bylaws to do away with them for the future.
> >
> > This is, in effect, another poll about the direction we're taking 2.x.
> > If we can't reach consensus on development directions without voting,
> > that's more evidence that the project should be split, IMO. -C
>
> +1e100
>
> Arun
>
> >
> > On Wed, May 15, 2013 at 2:21 PM, Steve Loughran <[hidden email]>
> wrote:
> >> On 15 May 2013 10:57, Arun C Murthy <[hidden email]> wrote:
> >>
> >>> Folks,
> >>>
> >>> A considerable number of people have expressed confusion regarding the
> >>> recent vote on 2.0.5, beta status etc. given lack of specifics, the
> voting
> >>> itself (validity of the vote itself, whose votes are binding) etc.
> >>>
> >>> IMHO technical arguments (incompatibility b/w 2.0 & 2.1, current
> stability
> >>> of 3 features under debate etc.) have been lost in the discussion in
> favor
> >>> of non-technical (almost dramatic) nuances such as "seizing the
> moment".
> >>> There is now dangerous talk of tolerating incompatibility b/w 2.0 and
> 2.1)
> >>> - this is a red flag for me; particularly when there are just 3
> features
> >>> being debated and active committers and contributors are confident of
> and
> >>> ready to stand by their work. All patches, I believe, are ready to be
> >>> merged in the the next few days per discussions on jira. This will,
> >>> clearly, not delay the other API work which everyone agrees is
> crucial. As
> >>> a result, I feel no recourse but to restart a new vote - all attempts
> at
> >>> calm, reasoned, civil discussion based on technical arguments have
> come to
> >>> naught - I apologize for the thrash caused to everyone's attention.
> >>>
> >>> To get past all of this confusion, I'd like to present an alternate,
> >>> specific proposal for consideration.
> >>>
> >>> I propose we continue the original plan and make a 2.0.5-beta release
> by
> >>> May end with the following content:
> >>> # HDFS-347
> >>> # HDFS Snapshots
> >>> # Windows support
> >>> # Necessary & final API/protocol changes such as:
> >>> * Final YARN API changes: YARN-386
> >>> * MR Binary Compatibility: MAPREDUCE-5108
> >>> * Final RPC cleanup: HADOOP-8990
> >>>
> >>> People working on the above features have all expressed considerable
> >>> comfort with them and are ready to stand-by to help expedite any
> necessary
> >>> bug-fixes etc. to get to stabilization quickly. I'm confident we can
> get
> >>> this release out by end of May. This sets stage for a hadoop-2.x GA
> release
> >>> right after with some more testing - this means I think I can quickly
> turn
> >>> around and make bug-fix releases as necessary right after 2.0.5-beta.
> >>>
> >>> I request that people consider helping out with this plan and sign up
> to
> >>> help push hadoop-2.x to stability as outlined above. I believe this
> will
> >>> help achieve our shared goals of quickly stabilizing hadoop-2 and help
> >>> ensure we can support it for forseeable future in a compatible manner
> for
> >>> the benefit of our users and downstream projects.
> >>>
> >>> Please vote, the vote will run the normal 7 days. Obviously, I'm +1.
> >>>
> >>
> >> +1 (binding)
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] - Release 2.0.5-beta

Andrew Purtell
In reply to this post by Arun Murthy
The other thread or "vote" or whatever at least served the purpose in fresh
surfacing of concerns. Talk of new features going in to a "beta" on a very
short short timetable is concerning for anyone with experience working on
large software projects. It's not a little ironic that this vote thread,
done in response to sort out the other one predicated on stability
concerns, begins with a laundry list of features and JIRAs to go in. I
think it is usually the case that a beta release receives only bugfixes*
over the alpha that proceeded it. This may just be a lack of consensus on
what "beta" means.

Please set aside discussion on particular features or Hadoop bylaws or
politics or debate club. I can't speak for all of downstream of course, but
to the extent that I can I can say we don't care about that. The core ask,
at least mine, is take a fresh look at reducing per-release disruptions to
the rest of the entire ecosystem that has grown up around Hadoop. Any
change made in Hadoop does not happen in isolation. On that other thread
were good suggestions on more frequent releases, reducing the change scope
per release, adopting merge windows, etc., all with the aim of allowing
downstream the time to sort out integration issues. The other comment on
this thread that suggests ASF governance structures being inadequate for
negotiating changes in a large ecosystem might be on to something, but at
the same time Apache BigTop may be an effective ASF-native answer to that.
Time will tell. It seems the tone of discussions between BigTop and Hadoop
in this thread are better than in the past. I will take further discussion
on volunteering resources to the BigTop lists.

* - I think this would include the great recent work in YARN on API
stability and in MR on restoring some backwards compatibility with MR1, but
perhaps these only meet my personal definition of bugfix, granted.



On Thu, May 16, 2013 at 5:20 AM, Arun C Murthy <[hidden email]> wrote:

> Great summary, thanks Vinod.
>
> On May 15, 2013, at 2:14 PM, Vinod Kumar Vavilapalli wrote:
>
> >
> > Roman, I keep this same argument again and again. Should've refuted
> earlier.
> >
> > Please list down all the issues that BigTop ran into *because of* new
> features. You continue to argue that new features are destabilizing 2.0.*,
> which I don't agree with at all. 2.0.3-alpha was the last time major
> features got merged in, and we found blockers irrespective of those.
> >
> > MAPREDUCE-5240 specifically isn't due to any feature merge. This was a
> bug. I'd say this is a long standing bug in 2.0.x. You sure this passed in
> 2.0.3? Even so, this is mostly broken by another bug-fix and *not* because
> of any feature.
> >
> > I quickly checked other bugs you reported in 2.0.x:
> > - MAPREDUCE-5088 was caused by the fix for HADOOP-9299 which was again a
> long standing issue in 2.0.x
> > - MAPREDUCE-3728 is similar
> > - MAPREDUCE-5117 is similar
> > - MAPREDUCE-4219 was a security related feature request from you.
> > - MAPREDUCE-3916 was because of new proxy-server added.
> >
> > I am not arguing that new features *may* destabilize the branch, but
> you've repeatedly stated this as if that were a fact.
> >
> > Really appreciate the testing done by BigTop, but please don't distort
> the facts.
> >
> > Thanks,
> > +Vinod
> >
> >
> > On May 15, 2013, at 1:29 PM, Roman Shaposhnik wrote:
> >
> >> Please tell me if my expectations are incorrect, but to me the -beta
> would
> >> signify it being a 'safe' target for the downstream components. We're
> still
> >> finding *very* basic and *very* disruptive issues (MAPREDUCE-5240 is
> >> a good example) that essentially mean DOA for downstream that depends
> >> on this functionality.
> >>
> >> Are we comfortable with delivering 2.0.5-beta and later on starting
> >> to discover things like MAPREDUCE-5240 more or less accidentally?
> >
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>


--
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] - Release 2.0.5-beta

Suresh Srinivas-2
On Wed, May 15, 2013 at 9:25 PM, Andrew Purtell <[hidden email]> wrote:

> The other thread or "vote" or whatever at least served the purpose in fresh
> surfacing of concerns. Talk of new features going in to a "beta" on a very
> short short timetable is concerning for anyone with experience working on
> large software projects. It's not a little ironic that this vote thread,
> done in response to sort out the other one predicated on stability
> concerns, begins with a laundry list of features and JIRAs to go in. I
> think it is usually the case that a beta release receives only bugfixes*
> over the alpha that proceeded it. This may just be a lack of consensus on
> what "beta" means.
>

Assuming that you are talking about HDFS features when you say
"features going into a beta on a very short short timetable" and
"laundry list" etc, I request you to take a cursory look at the development
of these features.

Snapshot is being developed since 2012 Nov, excluding the early
prototype that happened in 2012 May. Most of the development
was complete by the early February except for the support of rename
capability, which has been tricky. As regards to Windows support, this is a
work that has been happening for more than an year in many other branches.

So these features are not something that are impulsively developed
and irresponsibly pushed to a release. They have gone through
considerable testing and have been developed over a long time.

>
> Please set aside discussion on particular features or Hadoop bylaws or
> politics or debate club. I can't speak for all of downstream of course, but
> to the extent that I can I can say we don't care about that. The core ask,
> at least mine, is take a fresh look at reducing per-release disruptions to
> the rest of the entire ecosystem that has grown up around Hadoop.


What is the disruption you anticipate due to the current content of
the release?

If it is stability, I am confident that very few bugs will come out
of these features and stability should not be affected. This has been
the case for the HDFS features for many years. The development
is generally done in a feature branch, the feature is tested and stabilized
in that branch before merging to trunk. This is contrary to few people's
incorrect claims about how it has taken a long time to stabilize an HDFS
features in branch-2.

Needless to say stability is not just a concern of downstream projects. We
spend long hours, day in day out, trying to ensure features are stable as
core contributors.

Regards,
Suresh
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] - Release 2.0.5-beta

lohit
In reply to this post by Arun Murthy
Hi Arun,

Can we also include HDFS-3875 for this release.


2013/5/15 Arun C Murthy <[hidden email]>

> Folks,
>
> A considerable number of people have expressed confusion regarding the
> recent vote on 2.0.5, beta status etc. given lack of specifics, the voting
> itself (validity of the vote itself, whose votes are binding) etc.
>
> IMHO technical arguments (incompatibility b/w 2.0 & 2.1, current stability
> of 3 features under debate etc.) have been lost in the discussion in favor
> of non-technical (almost dramatic) nuances such as "seizing the moment".
> There is now dangerous talk of tolerating incompatibility b/w 2.0 and 2.1)
> - this is a red flag for me; particularly when there are just 3 features
> being debated and active committers and contributors are confident of and
> ready to stand by their work. All patches, I believe, are ready to be
> merged in the the next few days per discussions on jira. This will,
> clearly, not delay the other API work which everyone agrees is crucial. As
> a result, I feel no recourse but to restart a new vote - all attempts at
> calm, reasoned, civil discussion based on technical arguments have come to
> naught - I apologize for the thrash caused to everyone's attention.
>
> To get past all of this confusion, I'd like to present an alternate,
> specific proposal for consideration.
>
> I propose we continue the original plan and make a 2.0.5-beta release by
> May end with the following content:
> # HDFS-347
> # HDFS Snapshots
> # Windows support
> # Necessary & final API/protocol changes such as:
>  * Final YARN API changes: YARN-386
>  * MR Binary Compatibility: MAPREDUCE-5108
>  * Final RPC cleanup: HADOOP-8990
>
> People working on the above features have all expressed considerable
> comfort with them and are ready to stand-by to help expedite any necessary
> bug-fixes etc. to get to stabilization quickly. I'm confident we can get
> this release out by end of May. This sets stage for a hadoop-2.x GA release
> right after with some more testing - this means I think I can quickly turn
> around and make bug-fix releases as necessary right after 2.0.5-beta.
>
> I request that people consider helping out with this plan and sign up to
> help push hadoop-2.x to stability as outlined above. I believe this will
> help achieve our shared goals of quickly stabilizing hadoop-2 and help
> ensure we can support it for forseeable future in a compatible manner for
> the benefit of our users and downstream projects.
>
> Please vote, the vote will run the normal 7 days. Obviously, I'm +1.
>
> thanks,
> Arun
>
> PS: To keep this discussion grounded in technical details I've moved this
> to dev@ (bcc general@).
>
>


--
Have a Nice Day!
Lohit
1234