rough outline of where Solr's going

classic Classic list List threaded Threaded
44 messages Options
123
Reply | Threaded
Open this post in threaded view
|

rough outline of where Solr's going

Yonik Seeley-2
Here is a very rough list of what makes sense to me:
- since lucene is on a new major version, the next solr release
containing that sould have a new major version number
  - this does not preclude further releases on 1.x
  - for simplicity, and the "single dev" model, we should just sync
with lucene's... i.e. the next major Solr version would be 3.1
- branches/solr would become the new trunk, with a shared trunk with
lucene in some structure (see other thread)
- solr cloud branch gets merged in
- we move to Java6 (Java5 has already been EOLd by Sun unless you pay
money... and we need Java6 for zookeeper, scripting)
- remove deprecations (finally!), and perhaps some additional cleanups
that we've wanted to do

-Yonik
Reply | Threaded
Open this post in threaded view
|

Re: rough outline of where Solr's going

Yonik Seeley-2
another minor addition:
 - move to Junti4 for new tests... and some old tests might be
migrated (for speed issues)

I already have a SolrTestCaseJ4 that extends LuceneTestCase4J that
avoids spinning up a solr core for each test method... but I need to
be able to reference  LuceneTestCase4J from the lucene sources (i.e it
works in the IDE, but not on the command line right now).

-Yonik




On Tue, Mar 16, 2010 at 10:00 AM, Yonik Seeley <[hidden email]> wrote:

> Here is a very rough list of what makes sense to me:
> - since lucene is on a new major version, the next solr release
> containing that sould have a new major version number
>  - this does not preclude further releases on 1.x
>  - for simplicity, and the "single dev" model, we should just sync
> with lucene's... i.e. the next major Solr version would be 3.1
> - branches/solr would become the new trunk, with a shared trunk with
> lucene in some structure (see other thread)
> - solr cloud branch gets merged in
> - we move to Java6 (Java5 has already been EOLd by Sun unless you pay
> money... and we need Java6 for zookeeper, scripting)
> - remove deprecations (finally!), and perhaps some additional cleanups
> that we've wanted to do
>
> -Yonik
>
Reply | Threaded
Open this post in threaded view
|

Re: rough outline of where Solr's going

Grant Ingersoll-2
In reply to this post by Yonik Seeley-2

On Mar 16, 2010, at 11:00 AM, Yonik Seeley wrote:

> Here is a very rough list of what makes sense to me:
> - since lucene is on a new major version, the next solr release
> containing that sould have a new major version number
>  - this does not preclude further releases on 1.x
>  - for simplicity, and the "single dev" model, we should just sync
> with lucene's... i.e. the next major Solr version would be 3.1
> - branches/solr would become the new trunk, with a shared trunk with
> lucene in some structure (see other thread)
> - solr cloud branch gets merged in
> - we move to Java6 (Java5 has already been EOLd by Sun unless you pay
> money... and we need Java6 for zookeeper, scripting)

Hmm, how does that effect Lucene, though?  It is only on 1.5.

-Grant
Reply | Threaded
Open this post in threaded view
|

Re: rough outline of where Solr's going

Yonik Seeley
On Tue, Mar 16, 2010 at 10:45 AM, Grant Ingersoll <[hidden email]> wrote:

>
> On Mar 16, 2010, at 11:00 AM, Yonik Seeley wrote:
>
>> Here is a very rough list of what makes sense to me:
>> - since lucene is on a new major version, the next solr release
>> containing that sould have a new major version number
>>  - this does not preclude further releases on 1.x
>>  - for simplicity, and the "single dev" model, we should just sync
>> with lucene's... i.e. the next major Solr version would be 3.1
>> - branches/solr would become the new trunk, with a shared trunk with
>> lucene in some structure (see other thread)
>> - solr cloud branch gets merged in
>> - we move to Java6 (Java5 has already been EOLd by Sun unless you pay
>> money... and we need Java6 for zookeeper, scripting)
>
> Hmm, how does that effect Lucene, though?  It is only on 1.5.

Same way it did when lucene core was 1.4 but some of the contribs were 1.5
i.e. I don't think it really should affect anything.  Lucene core
moving to 1.5 is a different decision.

-Yonik
Reply | Threaded
Open this post in threaded view
|

Re: rough outline of where Solr's going

Yonik Seeley-2
In reply to this post by Yonik Seeley-2
One more addition:
 - Consider a different wiki... our current style will serve us poorly
across major version bumps esp.  We need versioning.   A different
option could include moving more documentation onto the website, where
it would be versioned.  Getting something easy to edit/change would be
the key there.... we don't have that currently.

-Yonik


On Tue, Mar 16, 2010 at 10:06 AM, Yonik Seeley <[hidden email]> wrote:

> another minor addition:
>  - move to Junti4 for new tests... and some old tests might be
> migrated (for speed issues)
>
> I already have a SolrTestCaseJ4 that extends LuceneTestCase4J that
> avoids spinning up a solr core for each test method... but I need to
> be able to reference  LuceneTestCase4J from the lucene sources (i.e it
> works in the IDE, but not on the command line right now).
>
> -Yonik
>
>
>
>
> On Tue, Mar 16, 2010 at 10:00 AM, Yonik Seeley <[hidden email]> wrote:
>> Here is a very rough list of what makes sense to me:
>> - since lucene is on a new major version, the next solr release
>> containing that sould have a new major version number
>>  - this does not preclude further releases on 1.x
>>  - for simplicity, and the "single dev" model, we should just sync
>> with lucene's... i.e. the next major Solr version would be 3.1
>> - branches/solr would become the new trunk, with a shared trunk with
>> lucene in some structure (see other thread)
>> - solr cloud branch gets merged in
>> - we move to Java6 (Java5 has already been EOLd by Sun unless you pay
>> money... and we need Java6 for zookeeper, scripting)
>> - remove deprecations (finally!), and perhaps some additional cleanups
>> that we've wanted to do
>>
>> -Yonik
>>
>
Reply | Threaded
Open this post in threaded view
|

Re: rough outline of where Solr's going

Bill Au
+1 on moving to Java 6 since Java 5 has been EOL'ed.

Bill

On Tue, Mar 16, 2010 at 12:00 PM, Yonik Seeley <[hidden email]> wrote:

> One more addition:
>  - Consider a different wiki... our current style will serve us poorly
> across major version bumps esp.  We need versioning.   A different
> option could include moving more documentation onto the website, where
> it would be versioned.  Getting something easy to edit/change would be
> the key there.... we don't have that currently.
>
> -Yonik
>
>
> On Tue, Mar 16, 2010 at 10:06 AM, Yonik Seeley <[hidden email]> wrote:
> > another minor addition:
> >  - move to Junti4 for new tests... and some old tests might be
> > migrated (for speed issues)
> >
> > I already have a SolrTestCaseJ4 that extends LuceneTestCase4J that
> > avoids spinning up a solr core for each test method... but I need to
> > be able to reference  LuceneTestCase4J from the lucene sources (i.e it
> > works in the IDE, but not on the command line right now).
> >
> > -Yonik
> >
> >
> >
> >
> > On Tue, Mar 16, 2010 at 10:00 AM, Yonik Seeley <[hidden email]> wrote:
> >> Here is a very rough list of what makes sense to me:
> >> - since lucene is on a new major version, the next solr release
> >> containing that sould have a new major version number
> >>  - this does not preclude further releases on 1.x
> >>  - for simplicity, and the "single dev" model, we should just sync
> >> with lucene's... i.e. the next major Solr version would be 3.1
> >> - branches/solr would become the new trunk, with a shared trunk with
> >> lucene in some structure (see other thread)
> >> - solr cloud branch gets merged in
> >> - we move to Java6 (Java5 has already been EOLd by Sun unless you pay
> >> money... and we need Java6 for zookeeper, scripting)
> >> - remove deprecations (finally!), and perhaps some additional cleanups
> >> that we've wanted to do
> >>
> >> -Yonik
> >>
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: rough outline of where Solr's going

hossman
In reply to this post by Yonik Seeley-2

: - since lucene is on a new major version, the next solr release
: containing that sould have a new major version number

This rationale concerns me less then making sure the version change
adequately communicates the significance of "upgrading' to users ... ie:
if most/many users will need to make config changes in order to upgrade,
that seems like a "major" upgrade to me, and the version number should
change in a "major" way ... if that means 2.0, 3.0, 10.0, or 3.1 i don't
care.

:   - for simplicity, and the "single dev" model, we should just sync
: with lucene's... i.e. the next major Solr version would be 3.1

this concerns me a little in that it doesn't feel like we've really
talked through how exactly we expect the version number game to play out
over time ... even if we get Lucene-Java and Solr onto the same
scheme now, we could easily find ourselves in a situation where
we're ready to release lucene-3.3 (ie: a minor release that is
back-compat with lucene-3.2 and addss some new features) but we also want
to release a new "major" version of solr that breaks compatibility with
solr-3.2 ... so what do we call that?

in short: trying to sync version numbers seems like a pain with very
little beenfit -- version numbers should communicate significance of
change, not dependency information.

: - remove deprecations (finally!), and perhaps some additional cleanups
: that we've wanted to do

As long as we bump the major version number to communicate the
significance, i'm all in favor of purging deprecations anytime people
want.


-Hoss

Reply | Threaded
Open this post in threaded view
|

Re: rough outline of where Solr's going

Kevin Osborn-2
I definitely agree with Chris here. Although Lucene and Solr are highly related, the version numbering should communicate whether Solr has changed in a significant or minor way to the end-user. A minor change in Lucene could cause major changes in Solr. And vice-versa, a major change in Lucene could actually result in very little change for the Solr end user.

-Kevin




________________________________
From: Chris Hostetter <[hidden email]>
To: [hidden email]
Sent: Tue, March 16, 2010 10:47:53 AM
Subject: Re: rough outline of where Solr's going


: - since lucene is on a new major version, the next solr release
: containing that sould have a new major version number

This rationale concerns me less then making sure the version change
adequately communicates the significance of "upgrading' to users ... ie:
if most/many users will need to make config changes in order to upgrade,
that seems like a "major" upgrade to me, and the version number should
change in a "major" way ... if that means 2.0, 3.0, 10.0, or 3.1 i don't
care.

:   - for simplicity, and the "single dev" model, we should just sync
: with lucene's... i.e. the next major Solr version would be 3.1

this concerns me a little in that it doesn't feel like we've really
talked through how exactly we expect the version number game to play out
over time ... even if we get Lucene-Java and Solr onto the same
scheme now, we could easily find ourselves in a situation where
we're ready to release lucene-3.3 (ie: a minor release that is
back-compat with lucene-3.2 and addss some new features) but we also want
to release a new "major" version of solr that breaks compatibility with
solr-3.2 ... so what do we call that?

in short: trying to sync version numbers seems like a pain with very
little beenfit -- version numbers should communicate significance of
change, not dependency information.

: - remove deprecations (finally!), and perhaps some additional cleanups
: that we've wanted to do

As long as we bump the major version number to communicate the
significance, i'm all in favor of purging deprecations anytime people
want.


-Hoss


     
Reply | Threaded
Open this post in threaded view
|

Re: rough outline of where Solr's going

Ted Dunning
The key word here is "end-user".

On Tue, Mar 16, 2010 at 10:57 AM, Kevin Osborn <[hidden email]> wrote:

> I definitely agree with Chris here. Although Lucene and Solr are highly
> related, the version numbering should communicate whether Solr has changed
> in a significant or minor way to the end-user. A minor change in Lucene
> could cause major changes in Solr. And vice-versa, a major change in Lucene
> could actually result in very little change for the Solr end user.
>
Reply | Threaded
Open this post in threaded view
|

Re: rough outline of where Solr's going

Yonik Seeley
In reply to this post by hossman
On Tue, Mar 16, 2010 at 1:47 PM, Chris Hostetter
<[hidden email]> wrote:
> even if we get Lucene-Java and Solr onto the same
> scheme now, we could easily find ourselves in a situation where
> we're ready to release lucene-3.3 (ie: a minor release that is
> back-compat with lucene-3.2 and addss some new features) but we also want
> to release a new "major" version of solr that breaks compatibility with
> solr-3.2 ... so what do we call that?

We try not to do that then.  Things make a lot more sense when one
starts thinking of them as a single project, w/o multiple downloads.

If major modules were to be pulled from Solr and put into Lucene, and
Solr wanted to make some big changes for a major version number bump?
How could it do so w/o lucene doing that?  It's the same issue.

-Yonik
Reply | Threaded
Open this post in threaded view
|

Re: rough outline of where Solr's going

hossman

: We try not to do that then.  Things make a lot more sense when one
: starts thinking of them as a single project, w/o multiple downloads.
:
: If major modules were to be pulled from Solr and put into Lucene, and
: Solr wanted to make some big changes for a major version number bump?
: How could it do so w/o lucene doing that?  It's the same issue.

No, actaully it's the converse issue -- if a major piece moves from "solr"
to "core" and a *person* wanted to make a major change to that piece of
functionality that wasn't backwards compatible, then "core" would
certianly need to undergo a major version bump.

But the way that changes is made may not have any impact on how that
functionality is ultimately exposed in Solr -- the "core" user based is
all JAva developers who care a lot about the Java API -- the "solr" user
based are typically not Java users, and have very little concern for what
the Java internals look like -- in every release Solr has changed internal
APIs in a way that was deeemed small enough to be acceptable to the
(small) percentage of the user population that writes plugins, but those
same changes in Lucene-Java would have mandated a major version change.

The main issue i'm trying to raise is the opposite of your example: when a
committer wants to make a big change to solr (frontend) code which will
have a big impact on how users interact with Solr, but in no way shape or
form affects the lucene core Java API ... how does that fit into a "lock
step" version policy?




-Hoss

Reply | Threaded
Open this post in threaded view
|

Re: rough outline of where Solr's going

Yonik Seeley-2
On Tue, Mar 16, 2010 at 2:25 PM, Chris Hostetter
<[hidden email]> wrote:

>
> : We try not to do that then.  Things make a lot more sense when one
> : starts thinking of them as a single project, w/o multiple downloads.
> :
> : If major modules were to be pulled from Solr and put into Lucene, and
> : Solr wanted to make some big changes for a major version number bump?
> : How could it do so w/o lucene doing that?  It's the same issue.
>
> No, actaully it's the converse issue -- if a major piece moves from "solr"
> to "core" and a *person* wanted to make a major change to that piece of
> functionality that wasn't backwards compatible, then "core" would
> certianly need to undergo a major version bump.

To try and put it simply - w/o an attempt at synchronized releases,
I'm against of code/modules moving out of solr.  It get's to the heart
of why we merged in the first place.

-Yonik
Reply | Threaded
Open this post in threaded view
|

Re: rough outline of where Solr's going

hossman

: > No, actaully it's the converse issue -- if a major piece moves from "solr"
: > to "core" and a *person* wanted to make a major change to that piece of
: > functionality that wasn't backwards compatible, then "core" would
: > certianly need to undergo a major version bump.
:
: To try and put it simply - w/o an attempt at synchronized releases,
: I'm against of code/modules moving out of solr.  It get's to the heart
: of why we merged in the first place.

Hmmm, i'm not sure what to say about that.  My recollection of the
discussion was that almost everyone agreed that refactoring and reducing
code overlap was a good idea, but synchronized releases seemed to be the
biggest sticking point people had (isn't that why there was a second (or
maybe third?) "take" on the vote ... to remove that item?)

In any case: it's still orthogonal to the point i'm trying to make
 
Even if Lucene-Java, and Solr, and a bunch of new modules refactored out
of the two, all start getting lock step releases, the version labels we
put on those releasees shouldn't neccessarily be identical.

We could easily find ourselves in either of the following two situations
for any arbitrary release off the "trunk" (assume it's 2011-05-23, and the
last release of both Lucene-Java and Solr was known as version "4.2")

1) Lucene-Java has made some massive public API changes that included
removing some deprecated APIs because they were horribly broken and
could corrupt data - so people agree that the version number should be
bumped to "5.0"; Solr has never used the old buggy API, and does not
expose any of the new underlying API changes, and has had very few changes
since 4.2 ... reving up to "5.0" would give users the impression that
something significant had changed, when in fact very little has changed.

2) Solr has made some major front end API changes and removed some
deprecated RequestHandlers that were buggy, so Solr really needs to bump
the version number up to "5.0"; Lucene-Java has made some very minor
feature additions, and the trunk is 100% back compat with 4.2 -- calling
it "lucene-Java 5.0" would give people a missleading ipression that it was
not API compatible with the previous release.


My key point being: Version numbers should communicate the
significance in change to the *user* of the product, and the users of
Solr are differnet then the users of Lucene-Java, so even if the releases
happen in lock step, that doesn't mean the verion numbers should be in
lock step.

-Hoss

Reply | Threaded
Open this post in threaded view
|

Re: rough outline of where Solr's going

Yonik Seeley-2
On Wed, Mar 17, 2010 at 8:15 PM, Chris Hostetter
<[hidden email]> wrote:

>
> : > No, actaully it's the converse issue -- if a major piece moves from "solr"
> : > to "core" and a *person* wanted to make a major change to that piece of
> : > functionality that wasn't backwards compatible, then "core" would
> : > certianly need to undergo a major version bump.
> :
> : To try and put it simply - w/o an attempt at synchronized releases,
> : I'm against of code/modules moving out of solr.  It get's to the heart
> : of why we merged in the first place.
>
> Hmmm, i'm not sure what to say about that.  My recollection of the
> discussion was that almost everyone agreed that refactoring and reducing
> code overlap was a good idea, but synchronized releases seemed to be the
> biggest sticking point people had (isn't that why there was a second (or
> maybe third?) "take" on the vote ... to remove that item?)

Most people were on board with synchronized releases.
Some people were treating it like a hard'n'fast contract forever... so
Mike added a second vote that added an "out" to address that:
 * Release details will be decided by dev community, but, Lucene may
  release without Solr.

The meaning, which most of us took (and expressed in the first vote
thread), that the idea was that we would try to release at the same
time, but lucene could still release if solr was clearly not ready.
It certainly wouldn't be the norm though.

In the interest of moving forward, perhaps we should just focus on the
immediate next major release - 3.1.  What happens after can wait.  We
never planned for absolutely all the "what if's" in Solr before the
merge - I'm not sure why we would need to now.

-Yonik
Reply | Threaded
Open this post in threaded view
|

Re: rough outline of where Solr's going

hossman

: In the interest of moving forward, perhaps we should just focus on the
: immediate next major release - 3.1.  What happens after can wait.  We
: never planned for absolutely all the "what if's" in Solr before the
: merge - I'm not sure why we would need to now.

I suppose, but if we call the next solr release "Solr 3.1" it will set a
precedent that will likely be impossible to maintain in a sane manner.


-Hoss

Reply | Threaded
Open this post in threaded view
|

Re: rough outline of where Solr's going

Grant Ingersoll-2
I tend to agree w/ Hoss here.  I don't think we have to be the same version numbers and I don't think we absolutely have to do lockstep releases.  

On Mar 17, 2010, at 8:38 PM, Chris Hostetter wrote:

>
> : In the interest of moving forward, perhaps we should just focus on the
> : immediate next major release - 3.1.  What happens after can wait.  We
> : never planned for absolutely all the "what if's" in Solr before the
> : merge - I'm not sure why we would need to now.
>
> I suppose, but if we call the next solr release "Solr 3.1" it will set a
> precedent that will likely be impossible to maintain in a sane manner.
>
>
> -Hoss
>


Reply | Threaded
Open this post in threaded view
|

Re: rough outline of where Solr's going

Yonik Seeley
On Wed, Mar 17, 2010 at 9:16 PM, Grant Ingersoll <[hidden email]> wrote:
> I tend to agree w/ Hoss here.  I don't think we have to be the same version numbers and I don't think we absolutely have to do lockstep releases.

No one said "absolutely".

It's important to try and release at the same time.  Without that, it
makes a lot less sense for Solr to be on Lucene's trunk.  Many of us
believe this is important to try to do... so I guess we'll try to do
that, even if everyone isn't on board.

-Yonik
Reply | Threaded
Open this post in threaded view
|

Re: rough outline of where Solr's going

Mattmann, Chris A (3010)
In reply to this post by hossman
Hi All,

>
>
> : In the interest of moving forward, perhaps we should just focus on the
> : immediate next major release - 3.1.  What happens after can wait.  We
> : never planned for absolutely all the "what if's" in Solr before the
> : merge - I'm not sure why we would need to now.
>
> I suppose, but if we call the next solr release "Solr 3.1" it will set a
> precedent that will likely be impossible to maintain in a sane manner.
>

+1. Release versions should make sense.

The next release of Solr, if it's a major version increment (anecdotally
which should have technical justification for being so -- not saying it
doesn't, but would be good to say why the major version increment is
necessary), should be +1.0 on the current major version number, which is 1.x
+ 1.0 = 2.x.

I'd ask the question what happened to 1.5, or 1.6 (or anywhere in-between),
but that's another question.

Here's the link to the thread where we talked about this before:

http://www.mail-archive.com/solr-dev@.../msg21936.html

I guess what I'm saying is that 2.0 is a lot better than 3.2 (and [here's
where my opinion comes in] 1.5 isn't so bad either).

Cheers,
Chris



++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: [hidden email]
WWW:   http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++


Reply | Threaded
Open this post in threaded view
|

Re: rough outline of where Solr's going

Robert Muir
In reply to this post by hossman
On Wed, Mar 17, 2010 at 8:15 PM, Chris Hostetter
<[hidden email]> wrote:
>
> My key point being: Version numbers should communicate the
> significance in change to the *user* of the product, and the users of
> Solr are differnet then the users of Lucene-Java, so even if the releases
> happen in lock step, that doesn't mean the verion numbers should be in
> lock step.
>

As you stated modules were important to think about for svn location,
then it would only make sense that they are important to think about
for release numbering, too.

So lets say we spin off a lucene-analyzers module, it should be 3.1,
too, as its already a "module" to some degree, and having a
lucene-analyzers-1.0.jar would be downright misleading.

So from this perspective of modules, with solr being a module
alongside lucene, 3.1 makes a lot of sense, and it also makes sense to
try to release things together if possible so that users aren't
confused.


--
Robert Muir
[hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: rough outline of where Solr's going

Grant Ingersoll-2
In reply to this post by Yonik Seeley

On Mar 17, 2010, at 9:41 PM, Yonik Seeley wrote:

> On Wed, Mar 17, 2010 at 9:16 PM, Grant Ingersoll <[hidden email]> wrote:
>> I tend to agree w/ Hoss here.  I don't think we have to be the same version numbers and I don't think we absolutely have to do lockstep releases.
>
> No one said "absolutely".
>
> It's important to try and release at the same time.  Without that, it
> makes a lot less sense for Solr to be on Lucene's trunk.  

I don't think releasing separately means Solr can't be on Lucene's trunk.  The two issues are orthogonal.

> Many of us
> believe this is important to try to do... so I guess we'll try to do
> that, even if everyone isn't on board.

I agree, it's important to try, but it isn't a show stopper, which is what the word lockstep implies.
123