Brainstorming on Improving the Release Process

[Grant Ingersoll-2]

(Long post, please bear with me and please read!)

Now that we have the release done (I'm working through the publication process now), I want to start the process of thinking about how we can improve the release process.  As I see it, building the artifacts and checking the legal items are now almost completely automated and testable at earlier stages in the game.

We have kept saying we want to release more often, but we have never defined actionable steps with which we can get there.  Goals without actionable steps are useless.

So, with that in mind, I'd like to brainstorm on how we can improve things a bit more.  Several us acted as RM this time around, so I think we have some common, shared knowledge to take advantage of this time as opposed to in the past where one person mostly just did the release in the background and then we all voted.

So, let's start with what we have right:

1. The Ant process for building a release candidate for both Lucene and Solr is almost identical now and fairly straightforward.
2. I think the feature freeze is a good thing, although it is a bit too long perhaps.
3. Pretty good documentation on the steps involved to branch, etc.
4. The new license validation stuff is a start for enforcing licensing up front more effectively.  What else can we validate up front in terms of packaging?
5. We have an awesome test infrastructure now.  I think it is safe to say that this version of Lucene is easily the most tested version we have ever shipped.

Things I see that can be improved, and these are only suggestions:

1.  We need to define the Minimum Effective Dose (MED - http://gizmodo.com/#!5709902/4+hour-body-the-principle-of-the-minimum-effective-dose) for producing a quality release.  Nothing more, nothing less.  I think one of our biggest problems is we don't know when we are done.  It's this loosey-goosey "we all agree" notion, but that's silly.  It's software, we should be able to test almost all of the artifacts for certain attributes and then release when they pass.  If we get something wrong, put in a test for it in the next release.  The old saying about perfect being the enemy of great applies here.

In other words, we don't have well defined things that we all are looking for when vetting a release candidate, other than what the ASF requires.  Look at the last few vote threads or for any of the previous threads.  It's obvious that we have a large variety of people doing a large variety of things when it comes to testing the candidates.  For instance, I do the following:
  a. check sigs., md5 hashes, etc.
  b. run the demos,
  c. run the Solr example and index some content,
  d. check over the LICENSE, NOTICE, CHANGES files
  e. Check the overall packaging, etc. is reasonable
  f. I run them through my training code

Others clearly do many other things.  Many of you have your own benchmark tests you run, others read over every last bit of documentation others still put the RC into their own application and test it.   All of this is good, but the problem is it is not _shared_ until the actual RC is up and it is not repeatable (not that all of it can be).  If you have benchmark code/tests that your run on an RC that doesn't involve proprietary code, why isn't it donated to the project so that we can all use it?  That way we don't have to wait until your -1 at the 11th hour to realize the RC is not good.  I personally don't care whether it's python or perl or whatever.  Something that works is better than nothing.  For instance, right now some of the committers have an Apache Extras project going for benchmarking.  Can we get this running on ASF resources on a regular basis?  If it's a computing resource issue, let's go to Infrastructure and ask for resources.  Infrastructure has repeatedly said that if a project needs resources to put together a proposal of what you want.  I bet we could get budget to spin up an EC2 instance once a week, run those long running tests (Test2B and other benchmarks) and then report back.  All of that can be automated.

Also, please think hard about whether the things you test can be automated and built into our test suite or at least run nightly or something on Jenkins and then donating them.  I know reading documentation can't, but what else?  For instance, could we auto-generate the file formats documentation?

2. We should be running and testing the release packaging process more regularly.

3.  I had an epiphany this release and it came via Hoss on a non release related issue where, likely unbeknownst to him, he called me out for not being focused on the release (https://issues.apache.org/jira/browse/SOLR-2366?focusedCommentId=12996154&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12996154)  I have often been a passive observer when it comes to releasing other than testing the RC (either that or I am the RM, it seems)  It's easy to sit and complain about there not being more frequent releases, but what are you doing about it?  Why are we working on trunk when we are trying to get a release out?  Obviously you are all free to do as you wish, but then don't complain when there aren't more frequent releases.  Yes, I am calling you out just like Hoss called me out.  If you aren't sure how you can help, ask.  Assuming we can better document #1 above, helping should be a lot easier in the future.

Now, obviously, not everyone can be involved in the actual building of the RC, but that doesn't mean you can't go build it yourself and start testing on it before the Vote comes out.  And it also doesn't mean you can't take a hard look at the process and see if there aren't better ways to do it.  I also think if we all focus more and coordinate our testing by defining test plans up front, we could get releases out in far less time.

4. Are those last minute changes really blockers?  We are all perfectionists, I know.  Our guilt gets to us if we happen to know there is something "broken" and we don't fix it.  I think we should add a "Known Issues" section to CHANGES.txt and we should be a bit tougher about keeping out last minute changes.   I was also Guilty as Charged in this case on the highlighter issue that Mark and I were working on.  Thankfully, Robert set me right.  I think that if it is only likely to effect a very small percentage of users AND there is a workaround, then we should simply document it and fix it on trunk and in the branch but not in the release.   If there are too many of these, then we are not ready to release.

5.  Most of the steps in http://wiki.apache.org/lucene-java/ReleaseTodo under the Branching & Feature Freeze section can be automated.  Same with the Publication section.

So, what are some possible actionable steps we can take?

1. To address #1, we can publish a test plan just like you would do in your day job.  See what OpenNLP does: https://cwiki.apache.org/confluence/display/OPENNLP/TestPlan1.5.1.  It starts by everyone addressing #1 above on our Wiki.  See http://wiki.apache.org/lucene-java/TestPlans and http://wiki.apache.org/solr/TestPlans for a starting spot.

2. To address #2, we can setup a Jenkins job that runs the packaging process regularly and somehow figure out how to reliably test it.

3. Shrugs on #3.  No one can make you participate, but we can make you feel guilty just like Hoss made me feel guilty.  For PMC members specifically, however, _perhaps_ if you don't vote for two or three releases in a row, then you shouldn't be on the PMC.  Your job on the PMC primarily consists of two things: vote on committers, vote on releases.  I can see giving a pass once given life happens.  Two or three in a row means you likely consider yourself emeritus or you just like having the title but not having any of the responsibility.   At a minimum send in a +0 and let us know you have a pulse.

4. For #4, this is a collective commitment to make the release happen as quickly as possible while still being correct for most people in most situations.  Focus on the goal.

5. For #5, we just need some people with some Ant skills to make it happen.

6. If we want to release every 6 months, then let's setup some sort of reminder system that nags us as we get closer or something.  Let's also be honest with ourselves about what features are in the release and what are not.  We don't have to have jam everything in all the time.

OK, enough of my ideas.  And I do stress that they are only ideas/observations.  What do others think?  Several of you have lived the RM pain now, let's take a good hard look at what can be done better so that we can release more often.

-Grant
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

[Grant Ingersoll-2]

I'll also add:

We need to figure out a better approach for CHANGES.txt.  Step 4 of the Publishing process is a PITA.

On Mar 30, 2011, at 8:22 AM, Grant Ingersoll wrote:

> (Long post, please bear with me and please read!)
>
> Now that we have the release done (I'm working through the publication process now), I want to start the process of thinking about how we can improve the release process.  As I see it, building the artifacts and checking the legal items are now almost completely automated and testable at earlier stages in the game.
>
> We have kept saying we want to release more often, but we have never defined actionable steps with which we can get there.  Goals without actionable steps are useless.
>
> So, with that in mind, I'd like to brainstorm on how we can improve things a bit more.  Several us acted as RM this time around, so I think we have some common, shared knowledge to take advantage of this time as opposed to in the past where one person mostly just did the release in the background and then we all voted.
>
> So, let's start with what we have right:
>
> 1. The Ant process for building a release candidate for both Lucene and Solr is almost identical now and fairly straightforward.
> 2. I think the feature freeze is a good thing, although it is a bit too long perhaps.
> 3. Pretty good documentation on the steps involved to branch, etc.
> 4. The new license validation stuff is a start for enforcing licensing up front more effectively.  What else can we validate up front in terms of packaging?
> 5. We have an awesome test infrastructure now.  I think it is safe to say that this version of Lucene is easily the most tested version we have ever shipped.
>
> Things I see that can be improved, and these are only suggestions:
>
> 1.  We need to define the Minimum Effective Dose (MED - http://gizmodo.com/#!5709902/4+hour-body-the-principle-of-the-minimum-effective-dose) for producing a quality release.  Nothing more, nothing less.  I think one of our biggest problems is we don't know when we are done.  It's this loosey-goosey "we all agree" notion, but that's silly.  It's software, we should be able to test almost all of the artifacts for certain attributes and then release when they pass.  If we get something wrong, put in a test for it in the next release.  The old saying about perfect being the enemy of great applies here.
>
> In other words, we don't have well defined things that we all are looking for when vetting a release candidate, other than what the ASF requires.  Look at the last few vote threads or for any of the previous threads.  It's obvious that we have a large variety of people doing a large variety of things when it comes to testing the candidates.  For instance, I do the following:
>  a. check sigs., md5 hashes, etc.
>  b. run the demos,
>  c. run the Solr example and index some content,
>  d. check over the LICENSE, NOTICE, CHANGES files
>  e. Check the overall packaging, etc. is reasonable
>  f. I run them through my training code
>
> Others clearly do many other things.  Many of you have your own benchmark tests you run, others read over every last bit of documentation others still put the RC into their own application and test it.   All of this is good, but the problem is it is not _shared_ until the actual RC is up and it is not repeatable (not that all of it can be).  If you have benchmark code/tests that your run on an RC that doesn't involve proprietary code, why isn't it donated to the project so that we can all use it?  That way we don't have to wait until your -1 at the 11th hour to realize the RC is not good.  I personally don't care whether it's python or perl or whatever.  Something that works is better than nothing.  For instance, right now some of the committers have an Apache Extras project going for benchmarking.  Can we get this running on ASF resources on a regular basis?  If it's a computing resource issue, let's go to Infrastructure and ask for resources.  Infrastructure has repeatedly said that if a project needs resources to put together a proposal of what you want.  I bet we could get budget to spin up an EC2 instance once a week, run those long running tests (Test2B and other benchmarks) and then report back.  All of that can be automated.
>
> Also, please think hard about whether the things you test can be automated and built into our test suite or at least run nightly or something on Jenkins and then donating them.  I know reading documentation can't, but what else?  For instance, could we auto-generate the file formats documentation?
>
> 2. We should be running and testing the release packaging process more regularly.
>
> 3.  I had an epiphany this release and it came via Hoss on a non release related issue where, likely unbeknownst to him, he called me out for not being focused on the release (https://issues.apache.org/jira/browse/SOLR-2366?focusedCommentId=12996154&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12996154)  I have often been a passive observer when it comes to releasing other than testing the RC (either that or I am the RM, it seems)  It's easy to sit and complain about there not being more frequent releases, but what are you doing about it?  Why are we working on trunk when we are trying to get a release out?  Obviously you are all free to do as you wish, but then don't complain when there aren't more frequent releases.  Yes, I am calling you out just like Hoss called me out.  If you aren't sure how you can help, ask.  Assuming we can better document #1 above, helping should be a lot easier in the future.
>
> Now, obviously, not everyone can be involved in the actual building of the RC, but that doesn't mean you can't go build it yourself and start testing on it before the Vote comes out.  And it also doesn't mean you can't take a hard look at the process and see if there aren't better ways to do it.  I also think if we all focus more and coordinate our testing by defining test plans up front, we could get releases out in far less time.
>
> 4. Are those last minute changes really blockers?  We are all perfectionists, I know.  Our guilt gets to us if we happen to know there is something "broken" and we don't fix it.  I think we should add a "Known Issues" section to CHANGES.txt and we should be a bit tougher about keeping out last minute changes.   I was also Guilty as Charged in this case on the highlighter issue that Mark and I were working on.  Thankfully, Robert set me right.  I think that if it is only likely to effect a very small percentage of users AND there is a workaround, then we should simply document it and fix it on trunk and in the branch but not in the release.   If there are too many of these, then we are not ready to release.
>
> 5.  Most of the steps in http://wiki.apache.org/lucene-java/ReleaseTodo under the Branching & Feature Freeze section can be automated.  Same with the Publication section.
>
> So, what are some possible actionable steps we can take?
>
> 1. To address #1, we can publish a test plan just like you would do in your day job.  See what OpenNLP does: https://cwiki.apache.org/confluence/display/OPENNLP/TestPlan1.5.1.  It starts by everyone addressing #1 above on our Wiki.  See http://wiki.apache.org/lucene-java/TestPlans and http://wiki.apache.org/solr/TestPlans for a starting spot.
>
> 2. To address #2, we can setup a Jenkins job that runs the packaging process regularly and somehow figure out how to reliably test it.
>
> 3. Shrugs on #3.  No one can make you participate, but we can make you feel guilty just like Hoss made me feel guilty.  For PMC members specifically, however, _perhaps_ if you don't vote for two or three releases in a row, then you shouldn't be on the PMC.  Your job on the PMC primarily consists of two things: vote on committers, vote on releases.  I can see giving a pass once given life happens.  Two or three in a row means you likely consider yourself emeritus or you just like having the title but not having any of the responsibility.   At a minimum send in a +0 and let us know you have a pulse.
>
> 4. For #4, this is a collective commitment to make the release happen as quickly as possible while still being correct for most people in most situations.  Focus on the goal.
>
> 5. For #5, we just need some people with some Ant skills to make it happen.
>
> 6. If we want to release every 6 months, then let's setup some sort of reminder system that nags us as we get closer or something.  Let's also be honest with ourselves about what features are in the release and what are not.  We don't have to have jam everything in all the time.
>
> OK, enough of my ideas.  And I do stress that they are only ideas/observations.  What do others think?  Several of you have lived the RM pain now, let's take a good hard look at what can be done better so that we can release more often.
>
> -Grant

--------------------------
Grant Ingersoll
http://www.lucidimagination.com/

Search the Lucene ecosystem docs using Solr/Lucene:
http://www.lucidimagination.com/search


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

[karl.wright]

Hi Grant,

This is a great post.

I'm not a committer for Lucene or Solr, but I'm seriously thinking that much of what Lucene/Solr does right should be considered by the project I AM a committer for: ManifoldCF.

Key things I would add based on experience with commercial software development:

(A) Left to their own devices, releases almost always get "too big".  The temptation is to just keep adding stuff, which winds up causing a delay, which adds more pressure for more features to be added to the release, etc.  The only way to address this that I've found which works is the "train leaving the station" model, where the date gets set in advance, and decisions as to feature inclusion based on that date.  Practically speaking, this means extended periods of time where development is happening in trunk and only selected changes are being pulled up to the release branch.

(B) Corollary to the "train leaving the station" model is that any massive global changes must occur only towards the beginning of the cycle.  Changes added to the release later in the cycle must be less and less destabilizing.  This often involves significant tradeoffs of the "proper" way to do things vs. the "least risky" way to do things.

(C) Finally, the larger the release, the LONGER the release branch must be active.  If you intend to release 4.0 this year, you should probably create the release branch no later than May/June, given the size of the 4.0 release already.

I'm sure all of this is well known, but I thought I'd state it nonetheless.

Karl


-----Original Message-----
From: ext Grant Ingersoll [mailto:[hidden email]]
Sent: Wednesday, March 30, 2011 8:22 AM
To: [hidden email]
Subject: Brainstorming on Improving the Release Process

(Long post, please bear with me and please read!)

Now that we have the release done (I'm working through the publication process now), I want to start the process of thinking about how we can improve the release process.  As I see it, building the artifacts and checking the legal items are now almost completely automated and testable at earlier stages in the game.

We have kept saying we want to release more often, but we have never defined actionable steps with which we can get there.  Goals without actionable steps are useless.

So, with that in mind, I'd like to brainstorm on how we can improve things a bit more.  Several us acted as RM this time around, so I think we have some common, shared knowledge to take advantage of this time as opposed to in the past where one person mostly just did the release in the background and then we all voted.

So, let's start with what we have right:

1. The Ant process for building a release candidate for both Lucene and Solr is almost identical now and fairly straightforward.
2. I think the feature freeze is a good thing, although it is a bit too long perhaps.
3. Pretty good documentation on the steps involved to branch, etc.
4. The new license validation stuff is a start for enforcing licensing up front more effectively.  What else can we validate up front in terms of packaging?
5. We have an awesome test infrastructure now.  I think it is safe to say that this version of Lucene is easily the most tested version we have ever shipped.

Things I see that can be improved, and these are only suggestions:

1.  We need to define the Minimum Effective Dose (MED - http://gizmodo.com/#!5709902/4+hour-body-the-principle-of-the-minimum-effective-dose) for producing a quality release.  Nothing more, nothing less.  I think one of our biggest problems is we don't know when we are done.  It's this loosey-goosey "we all agree" notion, but that's silly.  It's software, we should be able to test almost all of the artifacts for certain attributes and then release when they pass.  If we get something wrong, put in a test for it in the next release.  The old saying about perfect being the enemy of great applies here.

In other words, we don't have well defined things that we all are looking for when vetting a release candidate, other than what the ASF requires.  Look at the last few vote threads or for any of the previous threads.  It's obvious that we have a large variety of people doing a large variety of things when it comes to testing the candidates.  For instance, I do the following:
  a. check sigs., md5 hashes, etc.
  b. run the demos,
  c. run the Solr example and index some content,
  d. check over the LICENSE, NOTICE, CHANGES files
  e. Check the overall packaging, etc. is reasonable
  f. I run them through my training code

Others clearly do many other things.  Many of you have your own benchmark tests you run, others read over every last bit of documentation others still put the RC into their own application and test it.   All of this is good, but the problem is it is not _shared_ until the actual RC is up and it is not repeatable (not that all of it can be).  If you have benchmark code/tests that your run on an RC that doesn't involve proprietary code, why isn't it donated to the project so that we can all use it?  That way we don't have to wait until your -1 at the 11th hour to realize the RC is not good.  I personally don't care whether it's python or perl or whatever.  Something that works is better than nothing.  For instance, right now some of the committers have an Apache Extras project going for benchmarking.  Can we get this running on ASF resources on a regular basis?  If it's a computing resource issue, let's go to Infrastructure and ask for resources.  Infrastructure has repeatedly said that if a project needs resources to put together a proposal of what you want.  I bet we could get budget to spin up an EC2 instance once a week, run those long running tests (Test2B and other benchmarks) and then report back.  All of that can be automated.

Also, please think hard about whether the things you test can be automated and built into our test suite or at least run nightly or something on Jenkins and then donating them.  I know reading documentation can't, but what else?  For instance, could we auto-generate the file formats documentation?

2. We should be running and testing the release packaging process more regularly.

3.  I had an epiphany this release and it came via Hoss on a non release related issue where, likely unbeknownst to him, he called me out for not being focused on the release (https://issues.apache.org/jira/browse/SOLR-2366?focusedCommentId=12996154&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12996154)  I have often been a passive observer when it comes to releasing other than testing the RC (either that or I am the RM, it seems)  It's easy to sit and complain about there not being more frequent releases, but what are you doing about it?  Why are we working on trunk when we are trying to get a release out?  Obviously you are all free to do as you wish, but then don't complain when there aren't more frequent releases.  Yes, I am calling you out just like Hoss called me out.  If you aren't sure how you can help, ask.  Assuming we can better document #1 above, helping should be a lot easier in the future.

Now, obviously, not everyone can be involved in the actual building of the RC, but that doesn't mean you can't go build it yourself and start testing on it before the Vote comes out.  And it also doesn't mean you can't take a hard look at the process and see if there aren't better ways to do it.  I also think if we all focus more and coordinate our testing by defining test plans up front, we could get releases out in far less time.

4. Are those last minute changes really blockers?  We are all perfectionists, I know.  Our guilt gets to us if we happen to know there is something "broken" and we don't fix it.  I think we should add a "Known Issues" section to CHANGES.txt and we should be a bit tougher about keeping out last minute changes.   I was also Guilty as Charged in this case on the highlighter issue that Mark and I were working on.  Thankfully, Robert set me right.  I think that if it is only likely to effect a very small percentage of users AND there is a workaround, then we should simply document it and fix it on trunk and in the branch but not in the release.   If there are too many of these, then we are not ready to release.

5.  Most of the steps in http://wiki.apache.org/lucene-java/ReleaseTodo under the Branching & Feature Freeze section can be automated.  Same with the Publication section.

So, what are some possible actionable steps we can take?

1. To address #1, we can publish a test plan just like you would do in your day job.  See what OpenNLP does: https://cwiki.apache.org/confluence/display/OPENNLP/TestPlan1.5.1.  It starts by everyone addressing #1 above on our Wiki.  See http://wiki.apache.org/lucene-java/TestPlans and http://wiki.apache.org/solr/TestPlans for a starting spot.

2. To address #2, we can setup a Jenkins job that runs the packaging process regularly and somehow figure out how to reliably test it.

3. Shrugs on #3.  No one can make you participate, but we can make you feel guilty just like Hoss made me feel guilty.  For PMC members specifically, however, _perhaps_ if you don't vote for two or three releases in a row, then you shouldn't be on the PMC.  Your job on the PMC primarily consists of two things: vote on committers, vote on releases.  I can see giving a pass once given life happens.  Two or three in a row means you likely consider yourself emeritus or you just like having the title but not having any of the responsibility.   At a minimum send in a +0 and let us know you have a pulse.

4. For #4, this is a collective commitment to make the release happen as quickly as possible while still being correct for most people in most situations.  Focus on the goal.

5. For #5, we just need some people with some Ant skills to make it happen.

6. If we want to release every 6 months, then let's setup some sort of reminder system that nags us as we get closer or something.  Let's also be honest with ourselves about what features are in the release and what are not.  We don't have to have jam everything in all the time.

OK, enough of my ideas.  And I do stress that they are only ideas/observations.  What do others think?  Several of you have lived the RM pain now, let's take a good hard look at what can be done better so that we can release more often.

-Grant
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

[Robert Muir]

On Wed, Mar 30, 2011 at 8:22 AM, Grant Ingersoll <[hidden email]> wrote:
> (Long post, please bear with me and please read!)
>
> Now that we have the release done (I'm working through the publication process now), I want to start the process of thinking about how we can improve the release process.  As I see it, building the artifacts and checking the legal items are now almost completely automated and testable at earlier stages in the game.
>

Thanks for writing this up. Here is my major beef with 2 concrete suggestions:

It seems the current process is that we all develop and develop and at
some point we agree we want to try to release. At this point its the
RM's job to "polish a turd", and no serious community participation
takes place until an RC is actually produced: so its a chicken-and-egg
thing, perhaps with the RM even declaring publicly 'i dont expect this
to actually pass, i'm just building this to make you guys look at it'.

I think its probably hard/impossible to force people to review this
stuff before an RC, for some reason a VOTE seems to be the only thing
for people to take it seriously.

But what we can do is ask ourselves, how did the codebase become a
turd in the first place? Because at one point we released off the code
and the packaging was correct, there weren't javadocs warnings, and
there weren't licensing issues, etc.

So I think an important step would be to try to make more of this
"continuous", in other words, we did all the work to fix up the
codebase to make it releasable, lets implement things to enforce it
stays this way. It seems we did this for some things (e.g. code
correctness with the unit tests and licensing with the license
checker) but there is more to do.

A. implement the hudson-patch capability to vote -1 on patches that
break things as soon as they go on the JIRA issues. this is really
early feedback and I think will go a long way.
B. increase the scope of our 'ant test'/hudson runs to check more
things. For example, it would be nice if they failed on javadocs
warnings. Its insane if you think about it: we go to a ton of effort
to implement really cruel and picky unit tests to verify the
correctness of our code, but you can almost break the packaging and
documentation completely and the build still passes.

Anyway, we spend a lot of time on trying to make our code correct, but
our build is a bit messy. I know if we look at the time we spend on
search performance and correctness, and applied even 1% of this effort
to our build system to make it fast, picky, and and cleaner, that we
would be in much better shape as a development team, with a faster
compile/test/debug cycle to boot... I think there is a lot of
low-hanging fruit here, and I think this thread has encouraged me to
revisit the build and try to straighten some of this out.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

[Grant Ingersoll-2]

On Mar 30, 2011, at 9:19 AM, Robert Muir wrote:

> On Wed, Mar 30, 2011 at 8:22 AM, Grant Ingersoll <[hidden email]> wrote:
>> (Long post, please bear with me and please read!)
>>
>> Now that we have the release done (I'm working through the publication process now), I want to start the process of thinking about how we can improve the release process.  As I see it, building the artifacts and checking the legal items are now almost completely automated and testable at earlier stages in the game.
>>
>
> Thanks for writing this up. Here is my major beef with 2 concrete suggestions:
>
> It seems the current process is that we all develop and develop and at
> some point we agree we want to try to release. At this point its the
> RM's job to "polish a turd", and no serious community participation
> takes place until an RC is actually produced: so its a chicken-and-egg
> thing, perhaps with the RM even declaring publicly 'i dont expect this
> to actually pass, i'm just building this to make you guys look at it'.
>
> I think its probably hard/impossible to force people to review this
> stuff before an RC, for some reason a VOTE seems to be the only thing
> for people to take it seriously.
>
> But what we can do is ask ourselves, how did the codebase become a
> turd in the first place? Because at one point we released off the code
> and the packaging was correct, there weren't javadocs warnings, and
> there weren't licensing issues, etc.
>
> So I think an important step would be to try to make more of this
> "continuous", in other words, we did all the work to fix up the
> codebase to make it releasable, lets implement things to enforce it
> stays this way. It seems we did this for some things (e.g. code
> correctness with the unit tests and licensing with the license
> checker) but there is more to do.
>
> A. implement the hudson-patch capability to vote -1 on patches that
> break things as soon as they go on the JIRA issues. this is really
> early feedback and I think will go a long way.

+1.  I asked on [hidden email] if there was any "standard" way of doing this, or if there is a place someone can point me at to get this going.


> B. increase the scope of our 'ant test'/hudson runs to check more
> things. For example, it would be nice if they failed on javadocs
> warnings. Its insane if you think about it: we go to a ton of effort
> to implement really cruel and picky unit tests to verify the
> correctness of our code, but you can almost break the packaging and
> documentation completely and the build still passes.

+1 on failing on javadocs.

Also, what about code coverage?  We run all this Clover stuff, but how do we incorporate that into our dev. cycle?

>
> Anyway, we spend a lot of time on trying to make our code correct, but
> our build is a bit messy. I know if we look at the time we spend on
> search performance and correctness, and applied even 1% of this effort
> to our build system to make it fast, picky, and and cleaner, that we
> would be in much better shape as a development team, with a faster
> compile/test/debug cycle to boot... I think there is a lot of
> low-hanging fruit here, and I think this thread has encouraged me to
> revisit the build and try to straighten some of this out.

Yeah, our build is a bit messy, lots of recursion.  I'm still not totally happy w/ how license checking is hooked in.



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

[Upayavira]

On Wed, 30 Mar 2011 12:00 -0400, "Grant Ingersoll" <[hidden email]>
wrote:

>
> On Mar 30, 2011, at 9:19 AM, Robert Muir wrote:
>
> > On Wed, Mar 30, 2011 at 8:22 AM, Grant Ingersoll <[hidden email]> wrote:
> >> (Long post, please bear with me and please read!)
> >>
> >> Now that we have the release done (I'm working through the publication process now), I want to start the process of thinking about how we can improve the release process.  As I see it, building the artifacts and checking the legal items are now almost completely automated and testable at earlier stages in the game.
> >>
> >
> > Thanks for writing this up. Here is my major beef with 2 concrete suggestions:
> >
> > It seems the current process is that we all develop and develop and at
> > some point we agree we want to try to release. At this point its the
> > RM's job to "polish a turd", and no serious community participation
> > takes place until an RC is actually produced: so its a chicken-and-egg
> > thing, perhaps with the RM even declaring publicly 'i dont expect this
> > to actually pass, i'm just building this to make you guys look at it'.
> >
> > I think its probably hard/impossible to force people to review this
> > stuff before an RC, for some reason a VOTE seems to be the only thing
> > for people to take it seriously.
> >
> > But what we can do is ask ourselves, how did the codebase become a
> > turd in the first place? Because at one point we released off the code
> > and the packaging was correct, there weren't javadocs warnings, and
> > there weren't licensing issues, etc.
> >
> > So I think an important step would be to try to make more of this
> > "continuous", in other words, we did all the work to fix up the
> > codebase to make it releasable, lets implement things to enforce it
> > stays this way. It seems we did this for some things (e.g. code
> > correctness with the unit tests and licensing with the license
> > checker) but there is more to do.
> >
> > A. implement the hudson-patch capability to vote -1 on patches that
> > break things as soon as they go on the JIRA issues. this is really
> > early feedback and I think will go a long way.
>
> +1.  I asked on [hidden email] if there was any "standard" way of doing this,
> or if there is a place someone can point me at to get this going.
>
>
> > B. increase the scope of our 'ant test'/hudson runs to check more
> > things. For example, it would be nice if they failed on javadocs
> > warnings. Its insane if you think about it: we go to a ton of effort
> > to implement really cruel and picky unit tests to verify the
> > correctness of our code, but you can almost break the packaging and
> > documentation completely and the build still passes.
>
> +1 on failing on javadocs.
>
> Also, what about code coverage?  We run all this Clover stuff, but how do
> we incorporate that into our dev. cycle?
>
> >
> > Anyway, we spend a lot of time on trying to make our code correct, but
> > our build is a bit messy. I know if we look at the time we spend on
> > search performance and correctness, and applied even 1% of this effort
> > to our build system to make it fast, picky, and and cleaner, that we
> > would be in much better shape as a development team, with a faster
> > compile/test/debug cycle to boot... I think there is a lot of
> > low-hanging fruit here, and I think this thread has encouraged me to
> > revisit the build and try to straighten some of this out.
>
> Yeah, our build is a bit messy, lots of recursion.  I'm still not totally
> happy w/ how license checking is hooked in.

Are you willing to say more? I have a little time, and have done a lot
of work with Ant. Maybe I could help.

Upayavira
---
Enterprise Search Consultant at Sourcesense UK,
Making Sense of Open Source


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

[Robert Muir]

On Thu, Mar 31, 2011 at 9:40 AM, Upayavira <[hidden email]> wrote:

> Are you willing to say more? I have a little time, and have done a lot
> of work with Ant. Maybe I could help.
>
> Upayavira

Thanks, there is some followup discussion on this JIRA issue:
https://issues.apache.org/jira/browse/SOLR-2002

The "prototype" patch I refer to in the comments where solr build
system is changed to extend lucene's is the latest _merged.patch on
the issue: https://issues.apache.org/jira/secure/attachment/12456811/SOLR-2002_merged.patch

(Additionally as sort of a followup there are more comments/ideas
about additional things we could do besides just refactoring the build
system to be faster and simpler)

As a first step I think the patch needs to be brought up to trunk (it
gets out of date fast). I mentioned on the issue we can simply create
a branch to make coordination easier. A branch might seem silly for a
thing like this, but it would at least allow us to work together and
people could contribute parts (e.g. PMD integration or something)
without having to juggle huge out of sync patches.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

[Grant Ingersoll-2]

Other things to add:

1. Managing our website is a big pain in the butt.  Why do we need to publish PDFs again?  We really need to get on the new CMS.
2. Copying/moving the artifacts to the release area could be automated, too

At the end of the day, #1 below is what strikes me as the biggest impediment to releases.

>
>
> -----Original Message-----
> From: ext Grant Ingersoll [mailto:[hidden email]]
> Sent: Wednesday, March 30, 2011 8:22 AM
> To: [hidden email]
> Subject: Brainstorming on Improving the Release Process
>
> (Long post, please bear with me and please read!)
>
> Now that we have the release done (I'm working through the publication process now), I want to start the process of thinking about how we can improve the release process.  As I see it, building the artifacts and checking the legal items are now almost completely automated and testable at earlier stages in the game.
>
> We have kept saying we want to release more often, but we have never defined actionable steps with which we can get there.  Goals without actionable steps are useless.
>
> So, with that in mind, I'd like to brainstorm on how we can improve things a bit more.  Several us acted as RM this time around, so I think we have some common, shared knowledge to take advantage of this time as opposed to in the past where one person mostly just did the release in the background and then we all voted.
>
> So, let's start with what we have right:
>
> 1. The Ant process for building a release candidate for both Lucene and Solr is almost identical now and fairly straightforward.
> 2. I think the feature freeze is a good thing, although it is a bit too long perhaps.
> 3. Pretty good documentation on the steps involved to branch, etc.
> 4. The new license validation stuff is a start for enforcing licensing up front more effectively.  What else can we validate up front in terms of packaging?
> 5. We have an awesome test infrastructure now.  I think it is safe to say that this version of Lucene is easily the most tested version we have ever shipped.
>
> Things I see that can be improved, and these are only suggestions:
>
> 1.  We need to define the Minimum Effective Dose (MED - http://gizmodo.com/#!5709902/4+hour-body-the-principle-of-the-minimum-effective-dose) for producing a quality release.  Nothing more, nothing less.  I think one of our biggest problems is we don't know when we are done.  It's this loosey-goosey "we all agree" notion, but that's silly.  It's software, we should be able to test almost all of the artifacts for certain attributes and then release when they pass.  If we get something wrong, put in a test for it in the next release.  The old saying about perfect being the enemy of great applies here.
>
> In other words, we don't have well defined things that we all are looking for when vetting a release candidate, other than what the ASF requires.  Look at the last few vote threads or for any of the previous threads.  It's obvious that we have a large variety of people doing a large variety of things when it comes to testing the candidates.  For instance, I do the following:
>  a. check sigs., md5 hashes, etc.
>  b. run the demos,
>  c. run the Solr example and index some content,
>  d. check over the LICENSE, NOTICE, CHANGES files
>  e. Check the overall packaging, etc. is reasonable
>  f. I run them through my training code
>
> Others clearly do many other things.  Many of you have your own benchmark tests you run, others read over every last bit of documentation others still put the RC into their own application and test it.   All of this is good, but the problem is it is not _shared_ until the actual RC is up and it is not repeatable (not that all of it can be).  If you have benchmark code/tests that your run on an RC that doesn't involve proprietary code, why isn't it donated to the project so that we can all use it?  That way we don't have to wait until your -1 at the 11th hour to realize the RC is not good.  I personally don't care whether it's python or perl or whatever.  Something that works is better than nothing.  For instance, right now some of the committers have an Apache Extras project going for benchmarking.  Can we get this running on ASF resources on a regular basis?  If it's a computing resource issue, let's go to Infrastructure and ask for resources.  Infrastructure has repeatedly said that if a project needs resources to put together a proposal of what you want.  I bet we could get budget to spin up an EC2 instance once a week, run those long running tests (Test2B and other benchmarks) and then report back.  All of that can be automated.
>
> Also, please think hard about whether the things you test can be automated and built into our test suite or at least run nightly or something on Jenkins and then donating them.  I know reading documentation can't, but what else?  For instance, could we auto-generate the file formats documentation?
>
> 2. We should be running and testing the release packaging process more regularly.
>
> 3.  I had an epiphany this release and it came via Hoss on a non release related issue where, likely unbeknownst to him, he called me out for not being focused on the release (https://issues.apache.org/jira/browse/SOLR-2366?focusedCommentId=12996154&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12996154)  I have often been a passive observer when it comes to releasing other than testing the RC (either that or I am the RM, it seems)  It's easy to sit and complain about there not being more frequent releases, but what are you doing about it?  Why are we working on trunk when we are trying to get a release out?  Obviously you are all free to do as you wish, but then don't complain when there aren't more frequent releases.  Yes, I am calling you out just like Hoss called me out.  If you aren't sure how you can help, ask.  Assuming we can better document #1 above, helping should be a lot easier in the future.
>
> Now, obviously, not everyone can be involved in the actual building of the RC, but that doesn't mean you can't go build it yourself and start testing on it before the Vote comes out.  And it also doesn't mean you can't take a hard look at the process and see if there aren't better ways to do it.  I also think if we all focus more and coordinate our testing by defining test plans up front, we could get releases out in far less time.
>
> 4. Are those last minute changes really blockers?  We are all perfectionists, I know.  Our guilt gets to us if we happen to know there is something "broken" and we don't fix it.  I think we should add a "Known Issues" section to CHANGES.txt and we should be a bit tougher about keeping out last minute changes.   I was also Guilty as Charged in this case on the highlighter issue that Mark and I were working on.  Thankfully, Robert set me right.  I think that if it is only likely to effect a very small percentage of users AND there is a workaround, then we should simply document it and fix it on trunk and in the branch but not in the release.   If there are too many of these, then we are not ready to release.
>
> 5.  Most of the steps in http://wiki.apache.org/lucene-java/ReleaseTodo under the Branching & Feature Freeze section can be automated.  Same with the Publication section.
>
> So, what are some possible actionable steps we can take?
>
> 1. To address #1, we can publish a test plan just like you would do in your day job.  See what OpenNLP does: https://cwiki.apache.org/confluence/display/OPENNLP/TestPlan1.5.1.  It starts by everyone addressing #1 above on our Wiki.  See http://wiki.apache.org/lucene-java/TestPlans and http://wiki.apache.org/solr/TestPlans for a starting spot.
>
> 2. To address #2, we can setup a Jenkins job that runs the packaging process regularly and somehow figure out how to reliably test it.
>
> 3. Shrugs on #3.  No one can make you participate, but we can make you feel guilty just like Hoss made me feel guilty.  For PMC members specifically, however, _perhaps_ if you don't vote for two or three releases in a row, then you shouldn't be on the PMC.  Your job on the PMC primarily consists of two things: vote on committers, vote on releases.  I can see giving a pass once given life happens.  Two or three in a row means you likely consider yourself emeritus or you just like having the title but not having any of the responsibility.   At a minimum send in a +0 and let us know you have a pulse.
>
> 4. For #4, this is a collective commitment to make the release happen as quickly as possible while still being correct for most people in most situations.  Focus on the goal.
>
> 5. For #5, we just need some people with some Ant skills to make it happen.
>
> 6. If we want to release every 6 months, then let's setup some sort of reminder system that nags us as we get closer or something.  Let's also be honest with ourselves about what features are in the release and what are not.  We don't have to have jam everything in all the time.
>
> OK, enough of my ideas.  And I do stress that they are only ideas/observations.  What do others think?  Several of you have lived the RM pain now, let's take a good hard look at what can be done better so that we can release more often.
>
> -Grant
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

--------------------------
Grant Ingersoll
http://www.lucidimagination.com/

Search the Lucene ecosystem docs using Solr/Lucene:
http://www.lucidimagination.com/search


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

[Marvin Humphrey]

On Thu, Mar 31, 2011 at 11:45:53AM -0400, Grant Ingersoll wrote:
> Why do we need to publish PDFs again?  

IIRC, publishing PDFs is the default in Forrest.  It might have been a passive
choice.

Marvin Humphrey


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

[Grant Ingersoll-2]

On Mar 31, 2011, at 11:51 AM, Marvin Humphrey wrote:

> On Thu, Mar 31, 2011 at 11:45:53AM -0400, Grant Ingersoll wrote:
>> Why do we need to publish PDFs again?  
>
> IIRC, publishing PDFs is the default in Forrest.  It might have been a passive
> choice.

Yeah, it is.  I know.  Just one more thing to worry about when it is broken.

I think we need to simplify across a lot of our processes and get back to what I said earlier Minimum Effective Dose when it comes to builds, releases, etc.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

[Upayavira]

On Thu, 31 Mar 2011 09:51 -0400, "Robert Muir" <[hidden email]> wrote:

> On Thu, Mar 31, 2011 at 9:40 AM, Upayavira <[hidden email]> wrote:
>
> > Are you willing to say more? I have a little time, and have done a lot
> > of work with Ant. Maybe I could help.
> >
> > Upayavira
>
> Thanks, there is some followup discussion on this JIRA issue:
> https://issues.apache.org/jira/browse/SOLR-2002
>
> The "prototype" patch I refer to in the comments where solr build
> system is changed to extend lucene's is the latest _merged.patch on
> the issue:
> https://issues.apache.org/jira/secure/attachment/12456811/SOLR-2002_merged.patch
>
> (Additionally as sort of a followup there are more comments/ideas
> about additional things we could do besides just refactoring the build
> system to be faster and simpler)
>
> As a first step I think the patch needs to be brought up to trunk (it
> gets out of date fast). I mentioned on the issue we can simply create
> a branch to make coordination easier. A branch might seem silly for a
> thing like this, but it would at least allow us to work together and
> people could contribute parts (e.g. PMD integration or something)
> without having to juggle huge out of sync patches.

Thx. I'll take a look in the (uk) morning.

Upayavira
---
Enterprise Search Consultant at Sourcesense UK,
Making Sense of Open Source


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]