Solr 1.3.0 Release Lessons Learned

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Solr 1.3.0 Release Lessons Learned

Grant Ingersoll-2
Hey Solr Devs,

So, 1.3.0 is out.  Whew!  I think I survived.  I hope y'all did too.  
At any rate, I promised Lars I would follow up on his comment: http://lucene.markmail.org/message/ynsnkigymbv7kfqn?q=%5BVOTE%5D+Solr+1%2E3 
, so here goes.

So, what are the lessons learned?  What can we do to improve Solr's  
process, if any?

I saw a few pain points that I think are easily addressed:

1. IMO, way too long between release of 1.2 and 1.3 (1 year, 3  
months).  Yes, releases cause everyone to pause and take stock, but  
they are worthwhile, and not just technically.  Many users only use  
releases.  Many people don't notice a project except when they see it  
via some PR.  Releasing more often can help attract more contributors/
users which should lead to a better Solr.  Additionally, I imagine  
some people upgrading from 1.2 to 1.3 are trying to swallow a pretty  
big pill of features.  Granted, things should be back-compat, but even  
that is hard to track when something is a 1+ year ago.  I'd suggest we  
shoot for every 6 mos. or so, and maybe even some bug fix releases  
more often.

2. Last minute changes.  The mutlicore changes 1 week before release  
were pretty tough to swallow.  Great job to those involved who took it  
on, but still, let's not do that again, eh?  One _suggestion_ is that  
we try to front-load big features.  Hard to do, but maybe the other  
approach is that if we are about to take on a big new feature, we  
consider what other big new features are already in Solr and then  
maybe consider publishing them first and holding off for the next  
version the new feature.  Another possibility on this is a slight  
relaxation in back-compatibility "policy" in that for big features, we  
reserve the right to alter them in a build version release.  The main  
thing that this addresses is a lot of people feel uncomfortable on  
trunk, so maybe it's a way of getting more eyeballs.  Of course, we do  
this already to some extent when we mark things as experimental, so  
maybe nothing to change here.  Just thinking out loud.

3. We need to keep better track of NOTICEs, headers and library  
stuff.  Yonik and others did a lot to get these up to date again.  I  
know I'm especially guilty of forgetting to put headers on.  You can  
now run ant rat-sources for help in identifying offending files.

Thoughts?  Any thing else to consider?

-Grant
Reply | Threaded
Open this post in threaded view
|

Re: Solr 1.3.0 Release Lessons Learned

Lars Kotthoff
> I'd suggest we shoot for every 6 mos. or so, and maybe even some bug fix
> releases  more often.

Maybe we should even do something where we have a "stable" and an "experimental"
release. Especially for big things like the distributed search stuff this will
potentially attract more people to test everything. It'd also be easier to do
something that's not backwards compatible because people can always revert back
to the "stable" release.

Lars
Reply | Threaded
Open this post in threaded view
|

Re: Solr 1.3.0 Release Lessons Learned

Jason Rutherglen
> where we have a "stable" and an "experimental"
> release.

Good idea.  Also may be good to nominate a release manager.  It seemed
like features were being thrown in constantly that were perhaps beyond
the intended scope (was there one?) of SOLR 1.3.  Probably next time,
maybe 2-3 large features and some bug fixes and then do a release.

The biggest new feature right now that affects customers and users is
distributed search with failover.  It is hard to tell a customer that
at any time the search servers could fail and there (i.e. the master
can replicate bad data) is not much that can be done about it.

On Mon, Sep 22, 2008 at 9:42 AM, Lars Kotthoff <[hidden email]> wrote:

>> I'd suggest we shoot for every 6 mos. or so, and maybe even some bug fix
>> releases  more often.
>
> Maybe we should even do something where we have a "stable" and an "experimental"
> release. Especially for big things like the distributed search stuff this will
> potentially attract more people to test everything. It'd also be easier to do
> something that's not backwards compatible because people can always revert back
> to the "stable" release.
>
> Lars
>
Reply | Threaded
Open this post in threaded view
|

Re: Solr 1.3.0 Release Lessons Learned

Grant Ingersoll-2

On Sep 22, 2008, at 11:37 AM, Jason Rutherglen wrote:

>> where we have a "stable" and an "experimental"
>> release.
>
> Good idea.  Also may be good to nominate a release manager.  It seemed
> like features were being thrown in constantly that were perhaps beyond
> the intended scope (was there one?) of SOLR 1.3.  Probably next time,
> maybe 2-3 large features and some bug fixes and then do a release.
>

I only have two data points on this that I can add.  On the Lucene  
Java side, it's always been a bit like herding cats.  It really is up  
to the community, and usually a few committers to push for a release.  
We have a tacit agreement that we would like to release every 4-6  
months.  If you look at the Solr archives, we would often saw people  
asking when is 1.3 going to be out and our response was usually  
something like "we're working on X", and X kept changing.  This isn't  
a bad thing, necessarily.  Open Source is hard to plan, you never know  
where some nice new idea is coming from, so it could be there is  
momentum towards a release and then bam, someone comes in w/ a big new  
bug or a big feature.  Still, we could be better about saying, OK,  
this is great, let's do a release in 2 weeks and then add this nice  
new feature.

On the flip side of my experience is Hadoop.  Y! has assigned a number  
of resources to it, a number of which are committers, and also others  
as support.  They have people providing management in JIRA, driving  
release planning, etc.  Subscribing to the dev list is darn near  
overwhelming.  They also have fairly well timed out releases w/ well  
enumerated list of features, fixes, etc.  In other words, it's more  
commercially driven.

Personally, I think we just need to have some sort of "verbal"  
agreement, as committers/contributors to work towards more timely,  
smaller releases.  I don't want to release just for the sake of  
releasing, but I also want to see incremental advancements available  
to more people more often.

-Grant
Reply | Threaded
Open this post in threaded view
|

Re: Solr 1.3.0 Release Lessons Learned

Shalin Shekhar Mangar
In reply to this post by Grant Ingersoll-2
On Mon, Sep 22, 2008 at 7:04 PM, Grant Ingersoll <[hidden email]>wrote:

> Hey Solr Devs,
>
> So, 1.3.0 is out.  Whew!  I think I survived.  I hope y'all did too.  At
> any rate, I promised Lars I would follow up on his comment:
> http://lucene.markmail.org/message/ynsnkigymbv7kfqn?q=%5BVOTE%5D+Solr+1%2E3,
> so here goes.
>

Thanks for taking the initiative here, Grant.


>
> So, what are the lessons learned?  What can we do to improve Solr's
> process, if any?
>
> I saw a few pain points that I think are easily addressed:
>
> 1. IMO, way too long between release of 1.2 and 1.3 (1 year, 3 months).
>  Yes, releases cause everyone to pause and take stock, but they are
> worthwhile, and not just technically.  Many users only use releases.  Many
> people don't notice a project except when they see it via some PR.
>  Releasing more often can help attract more contributors/users which should
> lead to a better Solr.  Additionally, I imagine some people upgrading from
> 1.2 to 1.3 are trying to swallow a pretty big pill of features.  Granted,
> things should be back-compat, but even that is hard to track when something
> is a 1+ year ago.  I'd suggest we shoot for every 6 mos. or so, and maybe
> even some bug fix releases more often.
>

I'd love to see more frequent releases and I'm more than happy to work
towards that. I'd prefer if we put an upper bound to releases rather than a
strict interval. As Jason suggested, we should plan releases according to
features. Consider replication (SOLR-561), it is getting near to a fully
baked patch and it'd be nice to make it available to users much before six
months.

I'd like to propose a more pro-active approach to release planning by the
community. At any given time, let's have two versions in JIRA. Only those
issues which a committer has assigned to himself should be in the first
un-released version. All unassigned issues must be kept in the second
un-released version. If a committer assigns and promotes an issue to the
first un-released version, he should feel confident enough to resolve the
issue one way or another within 3 months of the last release else he should
mark it for the second version. At any given time, anybody can call a vote
on releasing with the trunk features. If we feel confident enough and the
list of resolved issues substantial enough, we can work according to our
current way of release planning (deferring open issues, creating a branch,
prioritizing bugs, putting up an RC and then release).

The above strategy will ensure that we stay nimble and do frequent releases
without putting a lot of pressure on committers or the release manager.
Stable trunk features will not starve and large features will not delay
releases indefinitely. We will have an upper bound on release dates as well
as the flexibility to release when the community feels confident.

Let us mark large features and core changes appropriately, creating
additional versions in Jira as and when applicable. We always have the
flexibility to promote features to earlier release versions if we feel it is
getting matured enough.

Thoughts?


>
> 2. Last minute changes.  The mutlicore changes 1 week before release were
> pretty tough to swallow.  Great job to those involved who took it on, but
> still, let's not do that again, eh?  One _suggestion_ is that we try to
> front-load big features.  Hard to do, but maybe the other approach is that
> if we are about to take on a big new feature, we consider what other big new
> features are already in Solr and then maybe consider publishing them first
> and holding off for the next version the new feature.  Another possibility
> on this is a slight relaxation in back-compatibility "policy" in that for
> big features, we reserve the right to alter them in a build version release.
>  The main thing that this addresses is a lot of people feel uncomfortable on
> trunk, so maybe it's a way of getting more eyeballs.  Of course, we do this
> already to some extent when we mark things as experimental, so maybe nothing
> to change here.  Just thinking out loud.


+1 to avoid last minute changes. At one point, it seemed like we'll never
release 1.3

>
>
> 3. We need to keep better track of NOTICEs, headers and library stuff.
>  Yonik and others did a lot to get these up to date again.  I know I'm
> especially guilty of forgetting to put headers on.  You can now run ant
> rat-sources for help in identifying offending files.
>

I too am guilty on this front going as far as suggesting to release with the
stax libs unchanged. I promise to pay closer attention to these aspects.



--
Regards,
Shalin Shekhar Mangar.
Reply | Threaded
Open this post in threaded view
|

Re: Solr 1.3.0 Release Lessons Learned

Mike Klaas

On 22-Sep-08, at 10:34 AM, Shalin Shekhar Mangar wrote:

>
> I'd like to propose a more pro-active approach to release planning  
> by the
> community. At any given time, let's have two versions in JIRA. Only  
> those
> issues which a committer has assigned to himself should be in the  
> first
> un-released version. All unassigned issues must be kept in the second
> un-released version. If a committer assigns and promotes an issue to  
> the
> first un-released version, he should feel confident enough to  
> resolve the
> issue one way or another within 3 months of the last release else he  
> should
> mark it for the second version. At any given time, anybody can call  
> a vote
> on releasing with the trunk features. If we feel confident enough  
> and the
> list of resolved issues substantial enough, we can work according to  
> our
> current way of release planning (deferring open issues, creating a  
> branch,
> prioritizing bugs, putting up an RC and then release).

I think that this is the right approach, but I don't think that it  
needs to be that complicated.  For issues without the "expectation of  
completion" that you mention, it is fine to just not assign a version  
to the issue.  It _would_ be useful, OTOH, to have a 2.0 version in  
JIRA for issues we know won't be resolved back-compatibly.

-Mike
Reply | Threaded
Open this post in threaded view
|

Re: Solr 1.3.0 Release Lessons Learned

Otis Gospodnetic-2
In reply to this post by Grant Ingersoll-2
I agree with Mike.  The simpler you make it the higher the chances of the plan being followed.  I had to re-read the part about un-released versions.  Moreover, rigid rules work great when people doing the work can really spend quality time on the project.  This means that people working on Solr through/for their work are really the people who could stick to the plan and everyone else will do whatever is possible for that individual at a time.  Hadoop is a good example, and with a couple of people working on Solr full-time now, more structured approach might be possible.  That said, I'd be careful of not making things too "work-like" (deadlines, committments, etc.) or else you risk losing people who have enough deadlines and other pressures already.

I'm not sure about that experimental vs. stable release suggestion - I think it can be as simple as "treat the trunk as experimental/in development" (which it is!) and only releases are stable.  In other words, no need to change anything there IMHO.

Otis




----- Original Message ----

> From: Mike Klaas <[hidden email]>
> To: [hidden email]
> Sent: Monday, September 22, 2008 1:40:59 PM
> Subject: Re: Solr 1.3.0 Release Lessons Learned
>
>
> On 22-Sep-08, at 10:34 AM, Shalin Shekhar Mangar wrote:
> >
> > I'd like to propose a more pro-active approach to release planning  
> > by the
> > community. At any given time, let's have two versions in JIRA. Only  
> > those
> > issues which a committer has assigned to himself should be in the  
> > first
> > un-released version. All unassigned issues must be kept in the second
> > un-released version. If a committer assigns and promotes an issue to  
> > the
> > first un-released version, he should feel confident enough to  
> > resolve the
> > issue one way or another within 3 months of the last release else he  
> > should
> > mark it for the second version. At any given time, anybody can call  
> > a vote
> > on releasing with the trunk features. If we feel confident enough  
> > and the
> > list of resolved issues substantial enough, we can work according to  
> > our
> > current way of release planning (deferring open issues, creating a  
> > branch,
> > prioritizing bugs, putting up an RC and then release).
>
> I think that this is the right approach, but I don't think that it  
> needs to be that complicated.  For issues without the "expectation of  
> completion" that you mention, it is fine to just not assign a version  
> to the issue.  It _would_ be useful, OTOH, to have a 2.0 version in  
> JIRA for issues we know won't be resolved back-compatibly.
>
> -Mike

Reply | Threaded
Open this post in threaded view
|

Re: Solr 1.3.0 Release Lessons Learned

Shalin Shekhar Mangar
I guess I went a bit overboard with the plan :-)

Yes, I agree about the point on hard deadlines. However, I do feel that
marking an issue to the next immediate release should represent a priority
to the issue from us. The core of my suggestion is continuous release
planning to ensure releasing early and releasing often. It is also an
incentive to scope issues appropriately.

On Tue, Sep 23, 2008 at 12:08 AM, Otis Gospodnetic <
[hidden email]> wrote:

> I agree with Mike.  The simpler you make it the higher the chances of the
> plan being followed.  I had to re-read the part about un-released versions.
>  Moreover, rigid rules work great when people doing the work can really
> spend quality time on the project.  This means that people working on Solr
> through/for their work are really the people who could stick to the plan and
> everyone else will do whatever is possible for that individual at a time.
>  Hadoop is a good example, and with a couple of people working on Solr
> full-time now, more structured approach might be possible.  That said, I'd
> be careful of not making things too "work-like" (deadlines, committments,
> etc.) or else you risk losing people who have enough deadlines and other
> pressures already.
>
> I'm not sure about that experimental vs. stable release suggestion - I
> think it can be as simple as "treat the trunk as experimental/in
> development" (which it is!) and only releases are stable.  In other words,
> no need to change anything there IMHO.
>
> Otis
>
>
>
>
> ----- Original Message ----
> > From: Mike Klaas <[hidden email]>
> > To: [hidden email]
> > Sent: Monday, September 22, 2008 1:40:59 PM
> > Subject: Re: Solr 1.3.0 Release Lessons Learned
> >
> >
> > On 22-Sep-08, at 10:34 AM, Shalin Shekhar Mangar wrote:
> > >
> > > I'd like to propose a more pro-active approach to release planning
> > > by the
> > > community. At any given time, let's have two versions in JIRA. Only
> > > those
> > > issues which a committer has assigned to himself should be in the
> > > first
> > > un-released version. All unassigned issues must be kept in the second
> > > un-released version. If a committer assigns and promotes an issue to
> > > the
> > > first un-released version, he should feel confident enough to
> > > resolve the
> > > issue one way or another within 3 months of the last release else he
> > > should
> > > mark it for the second version. At any given time, anybody can call
> > > a vote
> > > on releasing with the trunk features. If we feel confident enough
> > > and the
> > > list of resolved issues substantial enough, we can work according to
> > > our
> > > current way of release planning (deferring open issues, creating a
> > > branch,
> > > prioritizing bugs, putting up an RC and then release).
> >
> > I think that this is the right approach, but I don't think that it
> > needs to be that complicated.  For issues without the "expectation of
> > completion" that you mention, it is fine to just not assign a version
> > to the issue.  It _would_ be useful, OTOH, to have a 2.0 version in
> > JIRA for issues we know won't be resolved back-compatibly.
> >
> > -Mike
>
>


--
Regards,
Shalin Shekhar Mangar.
Reply | Threaded
Open this post in threaded view
|

Re: Solr 1.3.0 Release Lessons Learned

Otis Gospodnetic-2
In reply to this post by Grant Ingersoll-2
Yes, I think more consistent use of "Fix for version" will already be a very good step forward.  We started using that more consistently only right before the release, I'd say.  That is one easy thing we can do and it will allow us to quickly tell us where we are, whether we have enough meat for the release, what else is in the queue, etc.  Personally, I think sticking with just better use of that + vote JIRA functionality would get us 90% there.


Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch



----- Original Message ----

> From: Shalin Shekhar Mangar <[hidden email]>
> To: [hidden email]
> Sent: Monday, September 22, 2008 3:33:53 PM
> Subject: Re: Solr 1.3.0 Release Lessons Learned
>
> I guess I went a bit overboard with the plan :-)
>
> Yes, I agree about the point on hard deadlines. However, I do feel that
> marking an issue to the next immediate release should represent a priority
> to the issue from us. The core of my suggestion is continuous release
> planning to ensure releasing early and releasing often. It is also an
> incentive to scope issues appropriately.
>
> On Tue, Sep 23, 2008 at 12:08 AM, Otis Gospodnetic <
> [hidden email]> wrote:
>
> > I agree with Mike.  The simpler you make it the higher the chances of the
> > plan being followed.  I had to re-read the part about un-released versions.
> >  Moreover, rigid rules work great when people doing the work can really
> > spend quality time on the project.  This means that people working on Solr
> > through/for their work are really the people who could stick to the plan and
> > everyone else will do whatever is possible for that individual at a time.
> >  Hadoop is a good example, and with a couple of people working on Solr
> > full-time now, more structured approach might be possible.  That said, I'd
> > be careful of not making things too "work-like" (deadlines, committments,
> > etc.) or else you risk losing people who have enough deadlines and other
> > pressures already.
> >
> > I'm not sure about that experimental vs. stable release suggestion - I
> > think it can be as simple as "treat the trunk as experimental/in
> > development" (which it is!) and only releases are stable.  In other words,
> > no need to change anything there IMHO.
> >
> > Otis
> >
> >
> >
> >
> > ----- Original Message ----
> > > From: Mike Klaas
> > > To: [hidden email]
> > > Sent: Monday, September 22, 2008 1:40:59 PM
> > > Subject: Re: Solr 1.3.0 Release Lessons Learned
> > >
> > >
> > > On 22-Sep-08, at 10:34 AM, Shalin Shekhar Mangar wrote:
> > > >
> > > > I'd like to propose a more pro-active approach to release planning
> > > > by the
> > > > community. At any given time, let's have two versions in JIRA. Only
> > > > those
> > > > issues which a committer has assigned to himself should be in the
> > > > first
> > > > un-released version. All unassigned issues must be kept in the second
> > > > un-released version. If a committer assigns and promotes an issue to
> > > > the
> > > > first un-released version, he should feel confident enough to
> > > > resolve the
> > > > issue one way or another within 3 months of the last release else he
> > > > should
> > > > mark it for the second version. At any given time, anybody can call
> > > > a vote
> > > > on releasing with the trunk features. If we feel confident enough
> > > > and the
> > > > list of resolved issues substantial enough, we can work according to
> > > > our
> > > > current way of release planning (deferring open issues, creating a
> > > > branch,
> > > > prioritizing bugs, putting up an RC and then release).
> > >
> > > I think that this is the right approach, but I don't think that it
> > > needs to be that complicated.  For issues without the "expectation of
> > > completion" that you mention, it is fine to just not assign a version
> > > to the issue.  It _would_ be useful, OTOH, to have a 2.0 version in
> > > JIRA for issues we know won't be resolved back-compatibly.
> > >
> > > -Mike
> >
> >
>
>
> --
> Regards,
> Shalin Shekhar Mangar.