Solr Alpha (EA) release of Reference Branch

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
38 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Re: Solr Alpha (EA) release of Reference Branch

Tomás Fernández Löbbe
> I think the Solr 4 alpha/beta situation was different -- it was not some fork a committer was maintaining; it was the master branch of its time, and it was destined to be the very next release, not some possible future release.

Agree, 4’s alpha/beta was a very different situation.

> I believe it has to be an official release to have enough credibility. People trust the Apache brand and the community. This will ensure that we get enough people to test this out. The very objective of this release is to get help from our users to uncover any bugs. Most big shops will not deploy unofficial releases in their prod/staging environments. We wish to tick all the boxes for our users

I think this is fooling ourselves/our users. They trust Apache releases because we take them seriously, because they have the community behind, etc. This is a release from a feature branch, so we have to be cautious and very upfront. While I hope many go and test it and maybe deploy it to some of their environments, I feel we need to be careful not to send the wrong message.

> I'm -0 on doing an official release for alpha software because it's unnecessary to achieve the goals and somewhat confusing

I’d say I’m -1 to make it an official release for those reasons.

I think we should try to merge into master (whenever you are comfortable? Does it need to wait until 9.0 is out? What's the main reason for that? can we split the parts that do need to wait vs the ones that don’t?) and then people can be encouraged to run a nightly version in their test environments to help debug possible instability. If we need to do alpha/beta versions from master then I think that would make more sense.



On Mon, Oct 5, 2020 at 12:30 PM David Smiley <[hidden email]> wrote:
Thanks so much for your responses Ishan... I'm getting much more information in this thread than my attempts to get questions answered on the JIRA issue months ago.  And especially,  thank you for volunteering for the difficult porting efforts!

Tomas said:
 I do agree with the previous comments that calling it "Solr 10" (even with the "-alpha") would confuse users, maybe use "reference"? or maybe something in reference to SOLR-14788?

I have the opposite opinion.  This word "reference" is baffling to me despite whatever Mark's explanation is.  I like the justification Ishan gave for 10-alpha and I don't think I could re-phrase his justification any better.  *If* the release was _not_ official (thus wouldn't show up in the usual places anyone would look for a release), I think it would alleviate that confusion concern even more, although I think "alpha" ought to be enough of a signal not to use it without digging deeper on what's going on.

Alex then Ishan said:
> Maybe we could release it to
> committers community first and dogfood it "internally"?

Alex: It is meaningless. Committers don't run large scale installations. We barely even have time to take care of running unit tests before destabilizing our builds. We are not the right audience. However, we all can anyway check out the branch and start playing with it, even without a release. There are orgs that don't want to install any code that wasn't officially released; this release is geared towards them (to help us test this at their scale).

I don't think it can be said what committers do and don't do with regards to running Solr.  All of us would answer this differently and at different points in time.  From time to time, though not at present, I've been well positioned to try out a new version of Solr in a stage/test environment to see how it goes.  (Putting on my Salesforce metaphorical hat...) Even though I'm not able to deploy it in a realistic way today, I'm able to run a battery of tests to see if one of the features we depend on have changed or is broken.  That's useful feedback to an alpha release!  And even though I'm saying I'm not well positioned to try out some new Solr release in a production-ish setting now, it's something I could make a good case for internally since upgrades take a lot of effort where I work.  It's in our interest for SolrCloud to be very stable (of course).

Regardless, I think what you're driving at Ishan is that you want an "official" release -- one that goes through the whole ceremony.  You believe that people would be more likely to use it.  I think all we need to do is announce (similar to a real release) that there is some unofficial alpha distribution and that we want to solicit your feedback -- basically, help us find bugs.  Definitely publish a Docker image BTW -- it's the best way to try out any software.  I'm -0 on doing an official release for alpha software because it's unnecessary to achieve the goals and somewhat confusing.  I think the Solr 4 alpha/beta situation was different -- it was not some fork a committer was maintaining; it was the master branch of its time, and it was destined to be the very next release, not some possible future release.

~ David Smiley
Apache Lucene/Solr Search Developer
Reply | Threaded
Open this post in threaded view
|

Re: Solr Alpha (EA) release of Reference Branch

Noble Paul നോബിള്‍  नोब्ळ्
In reply to this post by David Smiley
>I don't think it can be said what committers do and don't do with regards to running Solr.  All of us would answer this differently and at different points in time.  

" I have run it in one large cluster, so it is certified to be bug free/stable" I don't think it's a reasonable approach. We need as much feedback from our users because each of them stress Solr in a different way. This is not to suggest that committers are not doing testing or their tests are not valid. When I talk to the committers out here they say they do not see any performance stability issues at all. But, my client reports issues on a day to day basis. 



> Definitely publish a Docker image BTW -- it's the best way to try out any software. 

Docker is not a big requirement for large scale installations. Most of them already have their own install scripts. Availability of docker is not important for them. If a user is only encouraged to install Solr because of a docker image , most likely they are not running a large enough cluster



On Tue, Oct 6, 2020, 6:30 AM David Smiley <[hidden email]> wrote:
Thanks so much for your responses Ishan... I'm getting much more information in this thread than my attempts to get questions answered on the JIRA issue months ago.  And especially,  thank you for volunteering for the difficult porting efforts!

Tomas said:
 I do agree with the previous comments that calling it "Solr 10" (even with the "-alpha") would confuse users, maybe use "reference"? or maybe something in reference to SOLR-14788?

I have the opposite opinion.  This word "reference" is baffling to me despite whatever Mark's explanation is.  I like the justification Ishan gave for 10-alpha and I don't think I could re-phrase his justification any better.  *If* the release was _not_ official (thus wouldn't show up in the usual places anyone would look for a release), I think it would alleviate that confusion concern even more, although I think "alpha" ought to be enough of a signal not to use it without digging deeper on what's going on.

Alex then Ishan said:
> Maybe we could release it to
> committers community first and dogfood it "internally"?

Alex: It is meaningless. Committers don't run large scale installations. We barely even have time to take care of running unit tests before destabilizing our builds. We are not the right audience. However, we all can anyway check out the branch and start playing with it, even without a release. There are orgs that don't want to install any code that wasn't officially released; this release is geared towards them (to help us test this at their scale).

I don't think it can be said what committers do and don't do with regards to running Solr.  All of us would answer this differently and at different points in time.  From time to time, though not at present, I've been well positioned to try out a new version of Solr in a stage/test environment to see how it goes.  (Putting on my Salesforce metaphorical hat...) Even though I'm not able to deploy it in a realistic way today, I'm able to run a battery of tests to see if one of the features we depend on have changed or is broken.  That's useful feedback to an alpha release!  And even though I'm saying I'm not well positioned to try out some new Solr release in a production-ish setting now, it's something I could make a good case for internally since upgrades take a lot of effort where I work.  It's in our interest for SolrCloud to be very stable (of course).

Regardless, I think what you're driving at Ishan is that you want an "official" release -- one that goes through the whole ceremony.  You believe that people would be more likely to use it.  I think all we need to do is announce (similar to a real release) that there is some unofficial alpha distribution and that we want to solicit your feedback -- basically, help us find bugs.  Definitely publish a Docker image BTW -- it's the best way to try out any software.  I'm -0 on doing an official release for alpha software because it's unnecessary to achieve the goals and somewhat confusing.  I think the Solr 4 alpha/beta situation was different -- it was not some fork a committer was maintaining; it was the master branch of its time, and it was destined to be the very next release, not some possible future release.

~ David Smiley
Apache Lucene/Solr Search Developer
Reply | Threaded
Open this post in threaded view
|

Re: Solr Alpha (EA) release of Reference Branch

Tomás Fernández Löbbe
I was thinking (and I haven’t flushed it out completely but will throw the idea) that an alternative approach with this timeline could be to cut 9x branch around November/December? And then you could merge into master, it would have the latest  changes from master plus the ref branch changes. From there any nightly build could be use to help test/debug.

That said I don’t know for sure what are the changes in the branch that do not belong in 9. The problem with them being 10x only is that backports would potentially be more difficult for all the life of 9.

On Mon, Oct 5, 2020 at 4:54 PM Noble Paul <[hidden email]> wrote:
>I don't think it can be said what committers do and don't do with regards to running Solr.  All of us would answer this differently and at different points in time.  

" I have run it in one large cluster, so it is certified to be bug free/stable" I don't think it's a reasonable approach. We need as much feedback from our users because each of them stress Solr in a different way. This is not to suggest that committers are not doing testing or their tests are not valid. When I talk to the committers out here they say they do not see any performance stability issues at all. But, my client reports issues on a day to day basis. 



> Definitely publish a Docker image BTW -- it's the best way to try out any software. 

Docker is not a big requirement for large scale installations. Most of them already have their own install scripts. Availability of docker is not important for them. If a user is only encouraged to install Solr because of a docker image , most likely they are not running a large enough cluster



On Tue, Oct 6, 2020, 6:30 AM David Smiley <[hidden email]> wrote:
Thanks so much for your responses Ishan... I'm getting much more information in this thread than my attempts to get questions answered on the JIRA issue months ago.  And especially,  thank you for volunteering for the difficult porting efforts!

Tomas said:
 I do agree with the previous comments that calling it "Solr 10" (even with the "-alpha") would confuse users, maybe use "reference"? or maybe something in reference to SOLR-14788?

I have the opposite opinion.  This word "reference" is baffling to me despite whatever Mark's explanation is.  I like the justification Ishan gave for 10-alpha and I don't think I could re-phrase his justification any better.  *If* the release was _not_ official (thus wouldn't show up in the usual places anyone would look for a release), I think it would alleviate that confusion concern even more, although I think "alpha" ought to be enough of a signal not to use it without digging deeper on what's going on.

Alex then Ishan said:
> Maybe we could release it to
> committers community first and dogfood it "internally"?

Alex: It is meaningless. Committers don't run large scale installations. We barely even have time to take care of running unit tests before destabilizing our builds. We are not the right audience. However, we all can anyway check out the branch and start playing with it, even without a release. There are orgs that don't want to install any code that wasn't officially released; this release is geared towards them (to help us test this at their scale).

I don't think it can be said what committers do and don't do with regards to running Solr.  All of us would answer this differently and at different points in time.  From time to time, though not at present, I've been well positioned to try out a new version of Solr in a stage/test environment to see how it goes.  (Putting on my Salesforce metaphorical hat...) Even though I'm not able to deploy it in a realistic way today, I'm able to run a battery of tests to see if one of the features we depend on have changed or is broken.  That's useful feedback to an alpha release!  And even though I'm saying I'm not well positioned to try out some new Solr release in a production-ish setting now, it's something I could make a good case for internally since upgrades take a lot of effort where I work.  It's in our interest for SolrCloud to be very stable (of course).

Regardless, I think what you're driving at Ishan is that you want an "official" release -- one that goes through the whole ceremony.  You believe that people would be more likely to use it.  I think all we need to do is announce (similar to a real release) that there is some unofficial alpha distribution and that we want to solicit your feedback -- basically, help us find bugs.  Definitely publish a Docker image BTW -- it's the best way to try out any software.  I'm -0 on doing an official release for alpha software because it's unnecessary to achieve the goals and somewhat confusing.  I think the Solr 4 alpha/beta situation was different -- it was not some fork a committer was maintaining; it was the master branch of its time, and it was destined to be the very next release, not some possible future release.

~ David Smiley
Apache Lucene/Solr Search Developer
Reply | Threaded
Open this post in threaded view
|

Re: Solr Alpha (EA) release of Reference Branch

Ishan Chattopadhyaya
As I said, I'm *personally* not confident in putting such a big changeset into master that wasn't vetted in a real user environment widely. I have, in the past, done enough bad things to Solr (directly or indirectly), and I don't want to repeat the same. Also, I'll be very uncomfortable if someone else did so.

Having said this, if someone else wants to port the changes over to master *without first getting enough real world testing*, feel free to do so, and I can focus my efforts elsewhere.

On Tue, 6 Oct, 2020, 9:22 am Tomás Fernández Löbbe, <[hidden email]> wrote:
I was thinking (and I haven’t flushed it out completely but will throw the idea) that an alternative approach with this timeline could be to cut 9x branch around November/December? And then you could merge into master, it would have the latest  changes from master plus the ref branch changes. From there any nightly build could be use to help test/debug.

That said I don’t know for sure what are the changes in the branch that do not belong in 9. The problem with them being 10x only is that backports would potentially be more difficult for all the life of 9.

On Mon, Oct 5, 2020 at 4:54 PM Noble Paul <[hidden email]> wrote:
>I don't think it can be said what committers do and don't do with regards to running Solr.  All of us would answer this differently and at different points in time.  

" I have run it in one large cluster, so it is certified to be bug free/stable" I don't think it's a reasonable approach. We need as much feedback from our users because each of them stress Solr in a different way. This is not to suggest that committers are not doing testing or their tests are not valid. When I talk to the committers out here they say they do not see any performance stability issues at all. But, my client reports issues on a day to day basis. 



> Definitely publish a Docker image BTW -- it's the best way to try out any software. 

Docker is not a big requirement for large scale installations. Most of them already have their own install scripts. Availability of docker is not important for them. If a user is only encouraged to install Solr because of a docker image , most likely they are not running a large enough cluster



On Tue, Oct 6, 2020, 6:30 AM David Smiley <[hidden email]> wrote:
Thanks so much for your responses Ishan... I'm getting much more information in this thread than my attempts to get questions answered on the JIRA issue months ago.  And especially,  thank you for volunteering for the difficult porting efforts!

Tomas said:
 I do agree with the previous comments that calling it "Solr 10" (even with the "-alpha") would confuse users, maybe use "reference"? or maybe something in reference to SOLR-14788?

I have the opposite opinion.  This word "reference" is baffling to me despite whatever Mark's explanation is.  I like the justification Ishan gave for 10-alpha and I don't think I could re-phrase his justification any better.  *If* the release was _not_ official (thus wouldn't show up in the usual places anyone would look for a release), I think it would alleviate that confusion concern even more, although I think "alpha" ought to be enough of a signal not to use it without digging deeper on what's going on.

Alex then Ishan said:
> Maybe we could release it to
> committers community first and dogfood it "internally"?

Alex: It is meaningless. Committers don't run large scale installations. We barely even have time to take care of running unit tests before destabilizing our builds. We are not the right audience. However, we all can anyway check out the branch and start playing with it, even without a release. There are orgs that don't want to install any code that wasn't officially released; this release is geared towards them (to help us test this at their scale).

I don't think it can be said what committers do and don't do with regards to running Solr.  All of us would answer this differently and at different points in time.  From time to time, though not at present, I've been well positioned to try out a new version of Solr in a stage/test environment to see how it goes.  (Putting on my Salesforce metaphorical hat...) Even though I'm not able to deploy it in a realistic way today, I'm able to run a battery of tests to see if one of the features we depend on have changed or is broken.  That's useful feedback to an alpha release!  And even though I'm saying I'm not well positioned to try out some new Solr release in a production-ish setting now, it's something I could make a good case for internally since upgrades take a lot of effort where I work.  It's in our interest for SolrCloud to be very stable (of course).

Regardless, I think what you're driving at Ishan is that you want an "official" release -- one that goes through the whole ceremony.  You believe that people would be more likely to use it.  I think all we need to do is announce (similar to a real release) that there is some unofficial alpha distribution and that we want to solicit your feedback -- basically, help us find bugs.  Definitely publish a Docker image BTW -- it's the best way to try out any software.  I'm -0 on doing an official release for alpha software because it's unnecessary to achieve the goals and somewhat confusing.  I think the Solr 4 alpha/beta situation was different -- it was not some fork a committer was maintaining; it was the master branch of its time, and it was destined to be the very next release, not some possible future release.

~ David Smiley
Apache Lucene/Solr Search Developer
Reply | Threaded
Open this post in threaded view
|

Re: Solr Alpha (EA) release of Reference Branch

Noble Paul നോബിള്‍  नोब्ळ्
@Tomas Fernandez Lobbe
This is a very risky proposition. Let's say we cut 9x and now there is
 a new master taken from the reference branch. That master is now
going to be 10.0.

* We never get it to the hands of the users anytime soon. We will
never be able to reconcile these 2 branches
* The master & 9x branches will diverge so much that we cannot make
any meaningful commits to any of these branches

Reference branch will have all the features of Solr 9.0 because it is
forked from master (some latest changes are yet to be merged). So,
reference branch is 100% compatible with master. A user should be able
to do a drop in replacement between 9.0 & reference branch. It is also
possible to have  a cluster with nodes running 9x and reference
together( A rolling restart should be possible).
Let's take stock of what we have today.

Choice 1:
* Mark has made some significant changes to improve performance stability
* Mark has promised to make all tests pass over the next 2-3 weeks
* Ishan has promised to do the work to make a release possible
* Ishan has promised to port changes to master as soon as we get
enough user feedback. He has also expressed his reservations on doing
this before it is tested in the wild
* I promise to do code review & cleanup as much as possible. But I'm
hesitant to give a stamp of approval to make it THE official release

Choice: 2
* Tomas promise to port changes from the branch to master
* Tomas make a release

Choice 3:
* Status quo
* We throw away all the good work that is done & demotivate the developers
* Deprive users of all the promised goodness

I'm not sure what else is a choice and who is volunteering to work on
that. We need people who are willing to offer to get their hands
dirty. Personally, I'm totally against choice #3 and I'm willing to
collaborate with anyone who is has a plan to make all the goodness in
the branch available to our users

On Tue, Oct 6, 2020 at 6:39 PM Ishan Chattopadhyaya
<[hidden email]> wrote:

>
> As I said, I'm *personally* not confident in putting such a big changeset into master that wasn't vetted in a real user environment widely. I have, in the past, done enough bad things to Solr (directly or indirectly), and I don't want to repeat the same. Also, I'll be very uncomfortable if someone else did so.
>
> Having said this, if someone else wants to port the changes over to master *without first getting enough real world testing*, feel free to do so, and I can focus my efforts elsewhere.
>
> On Tue, 6 Oct, 2020, 9:22 am Tomás Fernández Löbbe, <[hidden email]> wrote:
>>
>> I was thinking (and I haven’t flushed it out completely but will throw the idea) that an alternative approach with this timeline could be to cut 9x branch around November/December? And then you could merge into master, it would have the latest  changes from master plus the ref branch changes. From there any nightly build could be use to help test/debug.
>>
>> That said I don’t know for sure what are the changes in the branch that do not belong in 9. The problem with them being 10x only is that backports would potentially be more difficult for all the life of 9.
>>
>> On Mon, Oct 5, 2020 at 4:54 PM Noble Paul <[hidden email]> wrote:
>>>
>>> >I don't think it can be said what committers do and don't do with regards to running Solr.  All of us would answer this differently and at different points in time.
>>>
>>> " I have run it in one large cluster, so it is certified to be bug free/stable" I don't think it's a reasonable approach. We need as much feedback from our users because each of them stress Solr in a different way. This is not to suggest that committers are not doing testing or their tests are not valid. When I talk to the committers out here they say they do not see any performance stability issues at all. But, my client reports issues on a day to day basis.
>>>
>>>
>>>
>>> > Definitely publish a Docker image BTW -- it's the best way to try out any software.
>>>
>>> Docker is not a big requirement for large scale installations. Most of them already have their own install scripts. Availability of docker is not important for them. If a user is only encouraged to install Solr because of a docker image , most likely they are not running a large enough cluster
>>>
>>>
>>>
>>> On Tue, Oct 6, 2020, 6:30 AM David Smiley <[hidden email]> wrote:
>>>>
>>>> Thanks so much for your responses Ishan... I'm getting much more information in this thread than my attempts to get questions answered on the JIRA issue months ago.  And especially,  thank you for volunteering for the difficult porting efforts!
>>>>
>>>> Tomas said:
>>>>>
>>>>>  I do agree with the previous comments that calling it "Solr 10" (even with the "-alpha") would confuse users, maybe use "reference"? or maybe something in reference to SOLR-14788?
>>>>
>>>>
>>>> I have the opposite opinion.  This word "reference" is baffling to me despite whatever Mark's explanation is.  I like the justification Ishan gave for 10-alpha and I don't think I could re-phrase his justification any better.  *If* the release was _not_ official (thus wouldn't show up in the usual places anyone would look for a release), I think it would alleviate that confusion concern even more, although I think "alpha" ought to be enough of a signal not to use it without digging deeper on what's going on.
>>>>
>>>> Alex then Ishan said:
>>>>>
>>>>> > Maybe we could release it to
>>>>> > committers community first and dogfood it "internally"?
>>>>>
>>>>> Alex: It is meaningless. Committers don't run large scale installations. We barely even have time to take care of running unit tests before destabilizing our builds. We are not the right audience. However, we all can anyway check out the branch and start playing with it, even without a release. There are orgs that don't want to install any code that wasn't officially released; this release is geared towards them (to help us test this at their scale).
>>>>
>>>>
>>>> I don't think it can be said what committers do and don't do with regards to running Solr.  All of us would answer this differently and at different points in time.  From time to time, though not at present, I've been well positioned to try out a new version of Solr in a stage/test environment to see how it goes.  (Putting on my Salesforce metaphorical hat...) Even though I'm not able to deploy it in a realistic way today, I'm able to run a battery of tests to see if one of the features we depend on have changed or is broken.  That's useful feedback to an alpha release!  And even though I'm saying I'm not well positioned to try out some new Solr release in a production-ish setting now, it's something I could make a good case for internally since upgrades take a lot of effort where I work.  It's in our interest for SolrCloud to be very stable (of course).
>>>>
>>>> Regardless, I think what you're driving at Ishan is that you want an "official" release -- one that goes through the whole ceremony.  You believe that people would be more likely to use it.  I think all we need to do is announce (similar to a real release) that there is some unofficial alpha distribution and that we want to solicit your feedback -- basically, help us find bugs.  Definitely publish a Docker image BTW -- it's the best way to try out any software.  I'm -0 on doing an official release for alpha software because it's unnecessary to achieve the goals and somewhat confusing.  I think the Solr 4 alpha/beta situation was different -- it was not some fork a committer was maintaining; it was the master branch of its time, and it was destined to be the very next release, not some possible future release.
>>>>
>>>> ~ David Smiley
>>>> Apache Lucene/Solr Search Developer
>>>> http://www.linkedin.com/in/davidwsmiley



--
-----------------------------------------------------
Noble Paul

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Solr Alpha (EA) release of Reference Branch

Ishan Chattopadhyaya
In reply to this post by Ishan Chattopadhyaya

> Docker is not a big requirement for large scale installations. Most of them already have their own install scripts. Availability of docker is not important for them. If a user is only encouraged to install Solr because of a docker image , most likely they are not running a large enough cluster

I disagree, Noble. Having a docker image us going to be useful to some clients, with complex usecases. Great point, David!

On Tue, 6 Oct, 2020, 1:09 pm Ishan Chattopadhyaya, <[hidden email]> wrote:
As I said, I'm *personally* not confident in putting such a big changeset into master that wasn't vetted in a real user environment widely. I have, in the past, done enough bad things to Solr (directly or indirectly), and I don't want to repeat the same. Also, I'll be very uncomfortable if someone else did so.

Having said this, if someone else wants to port the changes over to master *without first getting enough real world testing*, feel free to do so, and I can focus my efforts elsewhere.

On Tue, 6 Oct, 2020, 9:22 am Tomás Fernández Löbbe, <[hidden email]> wrote:
I was thinking (and I haven’t flushed it out completely but will throw the idea) that an alternative approach with this timeline could be to cut 9x branch around November/December? And then you could merge into master, it would have the latest  changes from master plus the ref branch changes. From there any nightly build could be use to help test/debug.

That said I don’t know for sure what are the changes in the branch that do not belong in 9. The problem with them being 10x only is that backports would potentially be more difficult for all the life of 9.

On Mon, Oct 5, 2020 at 4:54 PM Noble Paul <[hidden email]> wrote:
>I don't think it can be said what committers do and don't do with regards to running Solr.  All of us would answer this differently and at different points in time.  

" I have run it in one large cluster, so it is certified to be bug free/stable" I don't think it's a reasonable approach. We need as much feedback from our users because each of them stress Solr in a different way. This is not to suggest that committers are not doing testing or their tests are not valid. When I talk to the committers out here they say they do not see any performance stability issues at all. But, my client reports issues on a day to day basis. 



> Definitely publish a Docker image BTW -- it's the best way to try out any software. 

Docker is not a big requirement for large scale installations. Most of them already have their own install scripts. Availability of docker is not important for them. If a user is only encouraged to install Solr because of a docker image , most likely they are not running a large enough cluster



On Tue, Oct 6, 2020, 6:30 AM David Smiley <[hidden email]> wrote:
Thanks so much for your responses Ishan... I'm getting much more information in this thread than my attempts to get questions answered on the JIRA issue months ago.  And especially,  thank you for volunteering for the difficult porting efforts!

Tomas said:
 I do agree with the previous comments that calling it "Solr 10" (even with the "-alpha") would confuse users, maybe use "reference"? or maybe something in reference to SOLR-14788?

I have the opposite opinion.  This word "reference" is baffling to me despite whatever Mark's explanation is.  I like the justification Ishan gave for 10-alpha and I don't think I could re-phrase his justification any better.  *If* the release was _not_ official (thus wouldn't show up in the usual places anyone would look for a release), I think it would alleviate that confusion concern even more, although I think "alpha" ought to be enough of a signal not to use it without digging deeper on what's going on.

Alex then Ishan said:
> Maybe we could release it to
> committers community first and dogfood it "internally"?

Alex: It is meaningless. Committers don't run large scale installations. We barely even have time to take care of running unit tests before destabilizing our builds. We are not the right audience. However, we all can anyway check out the branch and start playing with it, even without a release. There are orgs that don't want to install any code that wasn't officially released; this release is geared towards them (to help us test this at their scale).

I don't think it can be said what committers do and don't do with regards to running Solr.  All of us would answer this differently and at different points in time.  From time to time, though not at present, I've been well positioned to try out a new version of Solr in a stage/test environment to see how it goes.  (Putting on my Salesforce metaphorical hat...) Even though I'm not able to deploy it in a realistic way today, I'm able to run a battery of tests to see if one of the features we depend on have changed or is broken.  That's useful feedback to an alpha release!  And even though I'm saying I'm not well positioned to try out some new Solr release in a production-ish setting now, it's something I could make a good case for internally since upgrades take a lot of effort where I work.  It's in our interest for SolrCloud to be very stable (of course).

Regardless, I think what you're driving at Ishan is that you want an "official" release -- one that goes through the whole ceremony.  You believe that people would be more likely to use it.  I think all we need to do is announce (similar to a real release) that there is some unofficial alpha distribution and that we want to solicit your feedback -- basically, help us find bugs.  Definitely publish a Docker image BTW -- it's the best way to try out any software.  I'm -0 on doing an official release for alpha software because it's unnecessary to achieve the goals and somewhat confusing.  I think the Solr 4 alpha/beta situation was different -- it was not some fork a committer was maintaining; it was the master branch of its time, and it was destined to be the very next release, not some possible future release.

~ David Smiley
Apache Lucene/Solr Search Developer
Reply | Threaded
Open this post in threaded view
|

Re: Solr Alpha (EA) release of Reference Branch

Noble Paul നോബിള്‍  नोब्ळ्
Yes, A docker image will definitely help. I wasn't trying to downplay that

On Tue, Oct 6, 2020 at 6:55 PM Ishan Chattopadhyaya
<[hidden email]> wrote:

>
>
> > Docker is not a big requirement for large scale installations. Most of them already have their own install scripts. Availability of docker is not important for them. If a user is only encouraged to install Solr because of a docker image , most likely they are not running a large enough cluster
>
> I disagree, Noble. Having a docker image us going to be useful to some clients, with complex usecases. Great point, David!
>
> On Tue, 6 Oct, 2020, 1:09 pm Ishan Chattopadhyaya, <[hidden email]> wrote:
>>
>> As I said, I'm *personally* not confident in putting such a big changeset into master that wasn't vetted in a real user environment widely. I have, in the past, done enough bad things to Solr (directly or indirectly), and I don't want to repeat the same. Also, I'll be very uncomfortable if someone else did so.
>>
>> Having said this, if someone else wants to port the changes over to master *without first getting enough real world testing*, feel free to do so, and I can focus my efforts elsewhere.
>>
>> On Tue, 6 Oct, 2020, 9:22 am Tomás Fernández Löbbe, <[hidden email]> wrote:
>>>
>>> I was thinking (and I haven’t flushed it out completely but will throw the idea) that an alternative approach with this timeline could be to cut 9x branch around November/December? And then you could merge into master, it would have the latest  changes from master plus the ref branch changes. From there any nightly build could be use to help test/debug.
>>>
>>> That said I don’t know for sure what are the changes in the branch that do not belong in 9. The problem with them being 10x only is that backports would potentially be more difficult for all the life of 9.
>>>
>>> On Mon, Oct 5, 2020 at 4:54 PM Noble Paul <[hidden email]> wrote:
>>>>
>>>> >I don't think it can be said what committers do and don't do with regards to running Solr.  All of us would answer this differently and at different points in time.
>>>>
>>>> " I have run it in one large cluster, so it is certified to be bug free/stable" I don't think it's a reasonable approach. We need as much feedback from our users because each of them stress Solr in a different way. This is not to suggest that committers are not doing testing or their tests are not valid. When I talk to the committers out here they say they do not see any performance stability issues at all. But, my client reports issues on a day to day basis.
>>>>
>>>>
>>>>
>>>> > Definitely publish a Docker image BTW -- it's the best way to try out any software.
>>>>
>>>> Docker is not a big requirement for large scale installations. Most of them already have their own install scripts. Availability of docker is not important for them. If a user is only encouraged to install Solr because of a docker image , most likely they are not running a large enough cluster
>>>>
>>>>
>>>>
>>>> On Tue, Oct 6, 2020, 6:30 AM David Smiley <[hidden email]> wrote:
>>>>>
>>>>> Thanks so much for your responses Ishan... I'm getting much more information in this thread than my attempts to get questions answered on the JIRA issue months ago.  And especially,  thank you for volunteering for the difficult porting efforts!
>>>>>
>>>>> Tomas said:
>>>>>>
>>>>>>  I do agree with the previous comments that calling it "Solr 10" (even with the "-alpha") would confuse users, maybe use "reference"? or maybe something in reference to SOLR-14788?
>>>>>
>>>>>
>>>>> I have the opposite opinion.  This word "reference" is baffling to me despite whatever Mark's explanation is.  I like the justification Ishan gave for 10-alpha and I don't think I could re-phrase his justification any better.  *If* the release was _not_ official (thus wouldn't show up in the usual places anyone would look for a release), I think it would alleviate that confusion concern even more, although I think "alpha" ought to be enough of a signal not to use it without digging deeper on what's going on.
>>>>>
>>>>> Alex then Ishan said:
>>>>>>
>>>>>> > Maybe we could release it to
>>>>>> > committers community first and dogfood it "internally"?
>>>>>>
>>>>>> Alex: It is meaningless. Committers don't run large scale installations. We barely even have time to take care of running unit tests before destabilizing our builds. We are not the right audience. However, we all can anyway check out the branch and start playing with it, even without a release. There are orgs that don't want to install any code that wasn't officially released; this release is geared towards them (to help us test this at their scale).
>>>>>
>>>>>
>>>>> I don't think it can be said what committers do and don't do with regards to running Solr.  All of us would answer this differently and at different points in time.  From time to time, though not at present, I've been well positioned to try out a new version of Solr in a stage/test environment to see how it goes.  (Putting on my Salesforce metaphorical hat...) Even though I'm not able to deploy it in a realistic way today, I'm able to run a battery of tests to see if one of the features we depend on have changed or is broken.  That's useful feedback to an alpha release!  And even though I'm saying I'm not well positioned to try out some new Solr release in a production-ish setting now, it's something I could make a good case for internally since upgrades take a lot of effort where I work.  It's in our interest for SolrCloud to be very stable (of course).
>>>>>
>>>>> Regardless, I think what you're driving at Ishan is that you want an "official" release -- one that goes through the whole ceremony.  You believe that people would be more likely to use it.  I think all we need to do is announce (similar to a real release) that there is some unofficial alpha distribution and that we want to solicit your feedback -- basically, help us find bugs.  Definitely publish a Docker image BTW -- it's the best way to try out any software.  I'm -0 on doing an official release for alpha software because it's unnecessary to achieve the goals and somewhat confusing.  I think the Solr 4 alpha/beta situation was different -- it was not some fork a committer was maintaining; it was the master branch of its time, and it was destined to be the very next release, not some possible future release.
>>>>>
>>>>> ~ David Smiley
>>>>> Apache Lucene/Solr Search Developer
>>>>> http://www.linkedin.com/in/davidwsmiley



--
-----------------------------------------------------
Noble Paul

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Solr Alpha (EA) release of Reference Branch

Ilan Ginzburg
Another option to integrate this work into the main code line would be to understand what changes have been made and where (Mark's descriptions in Slack go in the right way but are still too high level), and then port or even redo them in main, one by one.

I think the danger is high to treat this branch as a black box (or an "all or nothing"). Using the merging itself to change our understanding and increase our knowledge of what was done can greatly reduce the risk.

We do develop new features in Solr 9 without beta releasing them, so if we port Mark's improvements by small chunks (and maybe in the process decide that some should not be ported or not now) I don't see why this can't integrate to become like other improvements done to the code. If specific changes do require a beta release, do that release from master and pick the right moment. 

I'm willing to help and I believe others will too if the amount of work for contributing is reasonable (i.e. not a three months effort). This requires documenting the changes done in that branch, pointing to where these changes happened and then picking them up one by one and porting them more or less independently of each other. We might only port a subset of changes by the time 9.0 is released, that's fine we can continue in following releases. 

My 2 cents... 
Ilan

Le mar. 6 oct. 2020 à 09:56, Noble Paul <[hidden email]> a écrit :
Yes, A docker image will definitely help. I wasn't trying to downplay that

On Tue, Oct 6, 2020 at 6:55 PM Ishan Chattopadhyaya
<[hidden email]> wrote:
>
>
> > Docker is not a big requirement for large scale installations. Most of them already have their own install scripts. Availability of docker is not important for them. If a user is only encouraged to install Solr because of a docker image , most likely they are not running a large enough cluster
>
> I disagree, Noble. Having a docker image us going to be useful to some clients, with complex usecases. Great point, David!
>
> On Tue, 6 Oct, 2020, 1:09 pm Ishan Chattopadhyaya, <[hidden email]> wrote:
>>
>> As I said, I'm *personally* not confident in putting such a big changeset into master that wasn't vetted in a real user environment widely. I have, in the past, done enough bad things to Solr (directly or indirectly), and I don't want to repeat the same. Also, I'll be very uncomfortable if someone else did so.
>>
>> Having said this, if someone else wants to port the changes over to master *without first getting enough real world testing*, feel free to do so, and I can focus my efforts elsewhere.
>>
>> On Tue, 6 Oct, 2020, 9:22 am Tomás Fernández Löbbe, <[hidden email]> wrote:
>>>
>>> I was thinking (and I haven’t flushed it out completely but will throw the idea) that an alternative approach with this timeline could be to cut 9x branch around November/December? And then you could merge into master, it would have the latest  changes from master plus the ref branch changes. From there any nightly build could be use to help test/debug.
>>>
>>> That said I don’t know for sure what are the changes in the branch that do not belong in 9. The problem with them being 10x only is that backports would potentially be more difficult for all the life of 9.
>>>
>>> On Mon, Oct 5, 2020 at 4:54 PM Noble Paul <[hidden email]> wrote:
>>>>
>>>> >I don't think it can be said what committers do and don't do with regards to running Solr.  All of us would answer this differently and at different points in time.
>>>>
>>>> " I have run it in one large cluster, so it is certified to be bug free/stable" I don't think it's a reasonable approach. We need as much feedback from our users because each of them stress Solr in a different way. This is not to suggest that committers are not doing testing or their tests are not valid. When I talk to the committers out here they say they do not see any performance stability issues at all. But, my client reports issues on a day to day basis.
>>>>
>>>>
>>>>
>>>> > Definitely publish a Docker image BTW -- it's the best way to try out any software.
>>>>
>>>> Docker is not a big requirement for large scale installations. Most of them already have their own install scripts. Availability of docker is not important for them. If a user is only encouraged to install Solr because of a docker image , most likely they are not running a large enough cluster
>>>>
>>>>
>>>>
>>>> On Tue, Oct 6, 2020, 6:30 AM David Smiley <[hidden email]> wrote:
>>>>>
>>>>> Thanks so much for your responses Ishan... I'm getting much more information in this thread than my attempts to get questions answered on the JIRA issue months ago.  And especially,  thank you for volunteering for the difficult porting efforts!
>>>>>
>>>>> Tomas said:
>>>>>>
>>>>>>  I do agree with the previous comments that calling it "Solr 10" (even with the "-alpha") would confuse users, maybe use "reference"? or maybe something in reference to SOLR-14788?
>>>>>
>>>>>
>>>>> I have the opposite opinion.  This word "reference" is baffling to me despite whatever Mark's explanation is.  I like the justification Ishan gave for 10-alpha and I don't think I could re-phrase his justification any better.  *If* the release was _not_ official (thus wouldn't show up in the usual places anyone would look for a release), I think it would alleviate that confusion concern even more, although I think "alpha" ought to be enough of a signal not to use it without digging deeper on what's going on.
>>>>>
>>>>> Alex then Ishan said:
>>>>>>
>>>>>> > Maybe we could release it to
>>>>>> > committers community first and dogfood it "internally"?
>>>>>>
>>>>>> Alex: It is meaningless. Committers don't run large scale installations. We barely even have time to take care of running unit tests before destabilizing our builds. We are not the right audience. However, we all can anyway check out the branch and start playing with it, even without a release. There are orgs that don't want to install any code that wasn't officially released; this release is geared towards them (to help us test this at their scale).
>>>>>
>>>>>
>>>>> I don't think it can be said what committers do and don't do with regards to running Solr.  All of us would answer this differently and at different points in time.  From time to time, though not at present, I've been well positioned to try out a new version of Solr in a stage/test environment to see how it goes.  (Putting on my Salesforce metaphorical hat...) Even though I'm not able to deploy it in a realistic way today, I'm able to run a battery of tests to see if one of the features we depend on have changed or is broken.  That's useful feedback to an alpha release!  And even though I'm saying I'm not well positioned to try out some new Solr release in a production-ish setting now, it's something I could make a good case for internally since upgrades take a lot of effort where I work.  It's in our interest for SolrCloud to be very stable (of course).
>>>>>
>>>>> Regardless, I think what you're driving at Ishan is that you want an "official" release -- one that goes through the whole ceremony.  You believe that people would be more likely to use it.  I think all we need to do is announce (similar to a real release) that there is some unofficial alpha distribution and that we want to solicit your feedback -- basically, help us find bugs.  Definitely publish a Docker image BTW -- it's the best way to try out any software.  I'm -0 on doing an official release for alpha software because it's unnecessary to achieve the goals and somewhat confusing.  I think the Solr 4 alpha/beta situation was different -- it was not some fork a committer was maintaining; it was the master branch of its time, and it was destined to be the very next release, not some possible future release.
>>>>>
>>>>> ~ David Smiley
>>>>> Apache Lucene/Solr Search Developer
>>>>> http://www.linkedin.com/in/davidwsmiley



--
-----------------------------------------------------
Noble Paul

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Solr Alpha (EA) release of Reference Branch

Ishan Chattopadhyaya

I'm willing to help and I believe others will too if the amount of work for contributing is reasonable (i.e. not a three months effort). 

I looked into the possibility of doing so. To me, it seemed to be that it is very hard to do so: possibly 1 year project for me. Problem is that it is hard to pull out a particular class of improvements (say thread management improvement) and have all tests pass with it (because tests have gotten extensive improvements of their own) and also observe the effect of the improvement. IIUC, every improvement to Solr seemed to require many iterations to get the tests happy. I remember Mark telling me that it may not even be possible for him to do something like that (i.e. bring all changes into master as tiny pieces).

What I volunteered to do, however, is to decompose roughly all the general improvements into smaller, manageable commits. However, making sure all tests pass at every commit point is beyond my capability.

On Tue, 6 Oct, 2020, 3:10 pm Ilan Ginzburg, <[hidden email]> wrote:
Another option to integrate this work into the main code line would be to understand what changes have been made and where (Mark's descriptions in Slack go in the right way but are still too high level), and then port or even redo them in main, one by one.

I think the danger is high to treat this branch as a black box (or an "all or nothing"). Using the merging itself to change our understanding and increase our knowledge of what was done can greatly reduce the risk.

We do develop new features in Solr 9 without beta releasing them, so if we port Mark's improvements by small chunks (and maybe in the process decide that some should not be ported or not now) I don't see why this can't integrate to become like other improvements done to the code. If specific changes do require a beta release, do that release from master and pick the right moment. 

I'm willing to help and I believe others will too if the amount of work for contributing is reasonable (i.e. not a three months effort). This requires documenting the changes done in that branch, pointing to where these changes happened and then picking them up one by one and porting them more or less independently of each other. We might only port a subset of changes by the time 9.0 is released, that's fine we can continue in following releases. 

My 2 cents... 
Ilan

Le mar. 6 oct. 2020 à 09:56, Noble Paul <[hidden email]> a écrit :
Yes, A docker image will definitely help. I wasn't trying to downplay that

On Tue, Oct 6, 2020 at 6:55 PM Ishan Chattopadhyaya
<[hidden email]> wrote:
>
>
> > Docker is not a big requirement for large scale installations. Most of them already have their own install scripts. Availability of docker is not important for them. If a user is only encouraged to install Solr because of a docker image , most likely they are not running a large enough cluster
>
> I disagree, Noble. Having a docker image us going to be useful to some clients, with complex usecases. Great point, David!
>
> On Tue, 6 Oct, 2020, 1:09 pm Ishan Chattopadhyaya, <[hidden email]> wrote:
>>
>> As I said, I'm *personally* not confident in putting such a big changeset into master that wasn't vetted in a real user environment widely. I have, in the past, done enough bad things to Solr (directly or indirectly), and I don't want to repeat the same. Also, I'll be very uncomfortable if someone else did so.
>>
>> Having said this, if someone else wants to port the changes over to master *without first getting enough real world testing*, feel free to do so, and I can focus my efforts elsewhere.
>>
>> On Tue, 6 Oct, 2020, 9:22 am Tomás Fernández Löbbe, <[hidden email]> wrote:
>>>
>>> I was thinking (and I haven’t flushed it out completely but will throw the idea) that an alternative approach with this timeline could be to cut 9x branch around November/December? And then you could merge into master, it would have the latest  changes from master plus the ref branch changes. From there any nightly build could be use to help test/debug.
>>>
>>> That said I don’t know for sure what are the changes in the branch that do not belong in 9. The problem with them being 10x only is that backports would potentially be more difficult for all the life of 9.
>>>
>>> On Mon, Oct 5, 2020 at 4:54 PM Noble Paul <[hidden email]> wrote:
>>>>
>>>> >I don't think it can be said what committers do and don't do with regards to running Solr.  All of us would answer this differently and at different points in time.
>>>>
>>>> " I have run it in one large cluster, so it is certified to be bug free/stable" I don't think it's a reasonable approach. We need as much feedback from our users because each of them stress Solr in a different way. This is not to suggest that committers are not doing testing or their tests are not valid. When I talk to the committers out here they say they do not see any performance stability issues at all. But, my client reports issues on a day to day basis.
>>>>
>>>>
>>>>
>>>> > Definitely publish a Docker image BTW -- it's the best way to try out any software.
>>>>
>>>> Docker is not a big requirement for large scale installations. Most of them already have their own install scripts. Availability of docker is not important for them. If a user is only encouraged to install Solr because of a docker image , most likely they are not running a large enough cluster
>>>>
>>>>
>>>>
>>>> On Tue, Oct 6, 2020, 6:30 AM David Smiley <[hidden email]> wrote:
>>>>>
>>>>> Thanks so much for your responses Ishan... I'm getting much more information in this thread than my attempts to get questions answered on the JIRA issue months ago.  And especially,  thank you for volunteering for the difficult porting efforts!
>>>>>
>>>>> Tomas said:
>>>>>>
>>>>>>  I do agree with the previous comments that calling it "Solr 10" (even with the "-alpha") would confuse users, maybe use "reference"? or maybe something in reference to SOLR-14788?
>>>>>
>>>>>
>>>>> I have the opposite opinion.  This word "reference" is baffling to me despite whatever Mark's explanation is.  I like the justification Ishan gave for 10-alpha and I don't think I could re-phrase his justification any better.  *If* the release was _not_ official (thus wouldn't show up in the usual places anyone would look for a release), I think it would alleviate that confusion concern even more, although I think "alpha" ought to be enough of a signal not to use it without digging deeper on what's going on.
>>>>>
>>>>> Alex then Ishan said:
>>>>>>
>>>>>> > Maybe we could release it to
>>>>>> > committers community first and dogfood it "internally"?
>>>>>>
>>>>>> Alex: It is meaningless. Committers don't run large scale installations. We barely even have time to take care of running unit tests before destabilizing our builds. We are not the right audience. However, we all can anyway check out the branch and start playing with it, even without a release. There are orgs that don't want to install any code that wasn't officially released; this release is geared towards them (to help us test this at their scale).
>>>>>
>>>>>
>>>>> I don't think it can be said what committers do and don't do with regards to running Solr.  All of us would answer this differently and at different points in time.  From time to time, though not at present, I've been well positioned to try out a new version of Solr in a stage/test environment to see how it goes.  (Putting on my Salesforce metaphorical hat...) Even though I'm not able to deploy it in a realistic way today, I'm able to run a battery of tests to see if one of the features we depend on have changed or is broken.  That's useful feedback to an alpha release!  And even though I'm saying I'm not well positioned to try out some new Solr release in a production-ish setting now, it's something I could make a good case for internally since upgrades take a lot of effort where I work.  It's in our interest for SolrCloud to be very stable (of course).
>>>>>
>>>>> Regardless, I think what you're driving at Ishan is that you want an "official" release -- one that goes through the whole ceremony.  You believe that people would be more likely to use it.  I think all we need to do is announce (similar to a real release) that there is some unofficial alpha distribution and that we want to solicit your feedback -- basically, help us find bugs.  Definitely publish a Docker image BTW -- it's the best way to try out any software.  I'm -0 on doing an official release for alpha software because it's unnecessary to achieve the goals and somewhat confusing.  I think the Solr 4 alpha/beta situation was different -- it was not some fork a committer was maintaining; it was the master branch of its time, and it was destined to be the very next release, not some possible future release.
>>>>>
>>>>> ~ David Smiley
>>>>> Apache Lucene/Solr Search Developer
>>>>> http://www.linkedin.com/in/davidwsmiley



--
-----------------------------------------------------
Noble Paul

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Solr Alpha (EA) release of Reference Branch

Noble Paul നോബിള്‍  नोब्ळ्
> I think the danger is high to treat this branch as a black box (or an "all or nothing").

True Ilan.  Ideally, I would like a few of us to study the code &
start pulling in changes we are confident of (even to 8x branch, why
not). We cannot burden a single developer to do everything.

This cannot be a task just for one or 2 devs. We all will have to work
together to decompose the changes and digest them into master. I can
do my bit.

But, I'm sure we may hit a point where certain changes cannot be
isolated and absorbed. We will have to collectively make a call, how
to absorb them

On Tue, Oct 6, 2020 at 9:00 PM Ishan Chattopadhyaya
<[hidden email]> wrote:

>
>
> I'm willing to help and I believe others will too if the amount of work for contributing is reasonable (i.e. not a three months effort).
>
> I looked into the possibility of doing so. To me, it seemed to be that it is very hard to do so: possibly 1 year project for me. Problem is that it is hard to pull out a particular class of improvements (say thread management improvement) and have all tests pass with it (because tests have gotten extensive improvements of their own) and also observe the effect of the improvement. IIUC, every improvement to Solr seemed to require many iterations to get the tests happy. I remember Mark telling me that it may not even be possible for him to do something like that (i.e. bring all changes into master as tiny pieces).
>
> What I volunteered to do, however, is to decompose roughly all the general improvements into smaller, manageable commits. However, making sure all tests pass at every commit point is beyond my capability.
>
> On Tue, 6 Oct, 2020, 3:10 pm Ilan Ginzburg, <[hidden email]> wrote:
>>
>> Another option to integrate this work into the main code line would be to understand what changes have been made and where (Mark's descriptions in Slack go in the right way but are still too high level), and then port or even redo them in main, one by one.
>>
>> I think the danger is high to treat this branch as a black box (or an "all or nothing"). Using the merging itself to change our understanding and increase our knowledge of what was done can greatly reduce the risk.
>>
>> We do develop new features in Solr 9 without beta releasing them, so if we port Mark's improvements by small chunks (and maybe in the process decide that some should not be ported or not now) I don't see why this can't integrate to become like other improvements done to the code. If specific changes do require a beta release, do that release from master and pick the right moment.
>>
>> I'm willing to help and I believe others will too if the amount of work for contributing is reasonable (i.e. not a three months effort). This requires documenting the changes done in that branch, pointing to where these changes happened and then picking them up one by one and porting them more or less independently of each other. We might only port a subset of changes by the time 9.0 is released, that's fine we can continue in following releases.
>>
>> My 2 cents...
>> Ilan
>>
>> Le mar. 6 oct. 2020 à 09:56, Noble Paul <[hidden email]> a écrit :
>>>
>>> Yes, A docker image will definitely help. I wasn't trying to downplay that
>>>
>>> On Tue, Oct 6, 2020 at 6:55 PM Ishan Chattopadhyaya
>>> <[hidden email]> wrote:
>>> >
>>> >
>>> > > Docker is not a big requirement for large scale installations. Most of them already have their own install scripts. Availability of docker is not important for them. If a user is only encouraged to install Solr because of a docker image , most likely they are not running a large enough cluster
>>> >
>>> > I disagree, Noble. Having a docker image us going to be useful to some clients, with complex usecases. Great point, David!
>>> >
>>> > On Tue, 6 Oct, 2020, 1:09 pm Ishan Chattopadhyaya, <[hidden email]> wrote:
>>> >>
>>> >> As I said, I'm *personally* not confident in putting such a big changeset into master that wasn't vetted in a real user environment widely. I have, in the past, done enough bad things to Solr (directly or indirectly), and I don't want to repeat the same. Also, I'll be very uncomfortable if someone else did so.
>>> >>
>>> >> Having said this, if someone else wants to port the changes over to master *without first getting enough real world testing*, feel free to do so, and I can focus my efforts elsewhere.
>>> >>
>>> >> On Tue, 6 Oct, 2020, 9:22 am Tomás Fernández Löbbe, <[hidden email]> wrote:
>>> >>>
>>> >>> I was thinking (and I haven’t flushed it out completely but will throw the idea) that an alternative approach with this timeline could be to cut 9x branch around November/December? And then you could merge into master, it would have the latest  changes from master plus the ref branch changes. From there any nightly build could be use to help test/debug.
>>> >>>
>>> >>> That said I don’t know for sure what are the changes in the branch that do not belong in 9. The problem with them being 10x only is that backports would potentially be more difficult for all the life of 9.
>>> >>>
>>> >>> On Mon, Oct 5, 2020 at 4:54 PM Noble Paul <[hidden email]> wrote:
>>> >>>>
>>> >>>> >I don't think it can be said what committers do and don't do with regards to running Solr.  All of us would answer this differently and at different points in time.
>>> >>>>
>>> >>>> " I have run it in one large cluster, so it is certified to be bug free/stable" I don't think it's a reasonable approach. We need as much feedback from our users because each of them stress Solr in a different way. This is not to suggest that committers are not doing testing or their tests are not valid. When I talk to the committers out here they say they do not see any performance stability issues at all. But, my client reports issues on a day to day basis.
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> > Definitely publish a Docker image BTW -- it's the best way to try out any software.
>>> >>>>
>>> >>>> Docker is not a big requirement for large scale installations. Most of them already have their own install scripts. Availability of docker is not important for them. If a user is only encouraged to install Solr because of a docker image , most likely they are not running a large enough cluster
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> On Tue, Oct 6, 2020, 6:30 AM David Smiley <[hidden email]> wrote:
>>> >>>>>
>>> >>>>> Thanks so much for your responses Ishan... I'm getting much more information in this thread than my attempts to get questions answered on the JIRA issue months ago.  And especially,  thank you for volunteering for the difficult porting efforts!
>>> >>>>>
>>> >>>>> Tomas said:
>>> >>>>>>
>>> >>>>>>  I do agree with the previous comments that calling it "Solr 10" (even with the "-alpha") would confuse users, maybe use "reference"? or maybe something in reference to SOLR-14788?
>>> >>>>>
>>> >>>>>
>>> >>>>> I have the opposite opinion.  This word "reference" is baffling to me despite whatever Mark's explanation is.  I like the justification Ishan gave for 10-alpha and I don't think I could re-phrase his justification any better.  *If* the release was _not_ official (thus wouldn't show up in the usual places anyone would look for a release), I think it would alleviate that confusion concern even more, although I think "alpha" ought to be enough of a signal not to use it without digging deeper on what's going on.
>>> >>>>>
>>> >>>>> Alex then Ishan said:
>>> >>>>>>
>>> >>>>>> > Maybe we could release it to
>>> >>>>>> > committers community first and dogfood it "internally"?
>>> >>>>>>
>>> >>>>>> Alex: It is meaningless. Committers don't run large scale installations. We barely even have time to take care of running unit tests before destabilizing our builds. We are not the right audience. However, we all can anyway check out the branch and start playing with it, even without a release. There are orgs that don't want to install any code that wasn't officially released; this release is geared towards them (to help us test this at their scale).
>>> >>>>>
>>> >>>>>
>>> >>>>> I don't think it can be said what committers do and don't do with regards to running Solr.  All of us would answer this differently and at different points in time.  From time to time, though not at present, I've been well positioned to try out a new version of Solr in a stage/test environment to see how it goes.  (Putting on my Salesforce metaphorical hat...) Even though I'm not able to deploy it in a realistic way today, I'm able to run a battery of tests to see if one of the features we depend on have changed or is broken.  That's useful feedback to an alpha release!  And even though I'm saying I'm not well positioned to try out some new Solr release in a production-ish setting now, it's something I could make a good case for internally since upgrades take a lot of effort where I work.  It's in our interest for SolrCloud to be very stable (of course).
>>> >>>>>
>>> >>>>> Regardless, I think what you're driving at Ishan is that you want an "official" release -- one that goes through the whole ceremony.  You believe that people would be more likely to use it.  I think all we need to do is announce (similar to a real release) that there is some unofficial alpha distribution and that we want to solicit your feedback -- basically, help us find bugs.  Definitely publish a Docker image BTW -- it's the best way to try out any software.  I'm -0 on doing an official release for alpha software because it's unnecessary to achieve the goals and somewhat confusing.  I think the Solr 4 alpha/beta situation was different -- it was not some fork a committer was maintaining; it was the master branch of its time, and it was destined to be the very next release, not some possible future release.
>>> >>>>>
>>> >>>>> ~ David Smiley
>>> >>>>> Apache Lucene/Solr Search Developer
>>> >>>>> http://www.linkedin.com/in/davidwsmiley
>>>
>>>
>>>
>>> --
>>> -----------------------------------------------------
>>> Noble Paul
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: [hidden email]
>>> For additional commands, e-mail: [hidden email]
>>>


--
-----------------------------------------------------
Noble Paul

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Solr Alpha (EA) release of Reference Branch

Tomás Fernández Löbbe
> Let's say we cut 9x and now there is a new master taken from the reference branch.
I never said “make a new master”, I said merge changes in ref branch into master. If things are broken into pieces like Ishan is suggesting, those changes can be merged into 9.x too. I only suggested this because you felt unsure about merging to master now and I guess this is due to fear of introducing bugs so close to a potential 9.0 release, is that not right?


> We will never be able to reconcile these 2 branches
Sorry, but how is that different if we do an alpha release from the branch now? What would be the process after that? Let's say people don't find issues and we want to merge those changes, what’s the plan then?

> Choice 1:
I’m fine with choice 1 if that’s what you want, as long as it’s not an official release for the reasons stated above.


> I promise to do code review & cleanup as much as possible. But I'm hesitant to give a stamp of approval to make it THE official release
What do you mean? I thought this is what you were suggesting, make an official release from the reference_impl branch?


I think Ilan’s last email is on spot, and I agree 100% with what he can express much better than I can :)

> Mark's descriptions in Slack go in the right way but are still too high level
Can someone share those here? or in Jira?

On Tue, Oct 6, 2020 at 5:09 AM Noble Paul <[hidden email]> wrote:
> I think the danger is high to treat this branch as a black box (or an "all or nothing").

True Ilan.  Ideally, I would like a few of us to study the code &
start pulling in changes we are confident of (even to 8x branch, why
not). We cannot burden a single developer to do everything.

This cannot be a task just for one or 2 devs. We all will have to work
together to decompose the changes and digest them into master. I can
do my bit.

But, I'm sure we may hit a point where certain changes cannot be
isolated and absorbed. We will have to collectively make a call, how
to absorb them

On Tue, Oct 6, 2020 at 9:00 PM Ishan Chattopadhyaya
<[hidden email]> wrote:
>
>
> I'm willing to help and I believe others will too if the amount of work for contributing is reasonable (i.e. not a three months effort).
>
> I looked into the possibility of doing so. To me, it seemed to be that it is very hard to do so: possibly 1 year project for me. Problem is that it is hard to pull out a particular class of improvements (say thread management improvement) and have all tests pass with it (because tests have gotten extensive improvements of their own) and also observe the effect of the improvement. IIUC, every improvement to Solr seemed to require many iterations to get the tests happy. I remember Mark telling me that it may not even be possible for him to do something like that (i.e. bring all changes into master as tiny pieces).
>
> What I volunteered to do, however, is to decompose roughly all the general improvements into smaller, manageable commits. However, making sure all tests pass at every commit point is beyond my capability.
>
> On Tue, 6 Oct, 2020, 3:10 pm Ilan Ginzburg, <[hidden email]> wrote:
>>
>> Another option to integrate this work into the main code line would be to understand what changes have been made and where (Mark's descriptions in Slack go in the right way but are still too high level), and then port or even redo them in main, one by one.
>>
>> I think the danger is high to treat this branch as a black box (or an "all or nothing"). Using the merging itself to change our understanding and increase our knowledge of what was done can greatly reduce the risk.
>>
>> We do develop new features in Solr 9 without beta releasing them, so if we port Mark's improvements by small chunks (and maybe in the process decide that some should not be ported or not now) I don't see why this can't integrate to become like other improvements done to the code. If specific changes do require a beta release, do that release from master and pick the right moment.
>>
>> I'm willing to help and I believe others will too if the amount of work for contributing is reasonable (i.e. not a three months effort). This requires documenting the changes done in that branch, pointing to where these changes happened and then picking them up one by one and porting them more or less independently of each other. We might only port a subset of changes by the time 9.0 is released, that's fine we can continue in following releases.
>>
>> My 2 cents...
>> Ilan
>>
>> Le mar. 6 oct. 2020 à 09:56, Noble Paul <[hidden email]> a écrit :
>>>
>>> Yes, A docker image will definitely help. I wasn't trying to downplay that
>>>
>>> On Tue, Oct 6, 2020 at 6:55 PM Ishan Chattopadhyaya
>>> <[hidden email]> wrote:
>>> >
>>> >
>>> > > Docker is not a big requirement for large scale installations. Most of them already have their own install scripts. Availability of docker is not important for them. If a user is only encouraged to install Solr because of a docker image , most likely they are not running a large enough cluster
>>> >
>>> > I disagree, Noble. Having a docker image us going to be useful to some clients, with complex usecases. Great point, David!
>>> >
>>> > On Tue, 6 Oct, 2020, 1:09 pm Ishan Chattopadhyaya, <[hidden email]> wrote:
>>> >>
>>> >> As I said, I'm *personally* not confident in putting such a big changeset into master that wasn't vetted in a real user environment widely. I have, in the past, done enough bad things to Solr (directly or indirectly), and I don't want to repeat the same. Also, I'll be very uncomfortable if someone else did so.
>>> >>
>>> >> Having said this, if someone else wants to port the changes over to master *without first getting enough real world testing*, feel free to do so, and I can focus my efforts elsewhere.
>>> >>
>>> >> On Tue, 6 Oct, 2020, 9:22 am Tomás Fernández Löbbe, <[hidden email]> wrote:
>>> >>>
>>> >>> I was thinking (and I haven’t flushed it out completely but will throw the idea) that an alternative approach with this timeline could be to cut 9x branch around November/December? And then you could merge into master, it would have the latest  changes from master plus the ref branch changes. From there any nightly build could be use to help test/debug.
>>> >>>
>>> >>> That said I don’t know for sure what are the changes in the branch that do not belong in 9. The problem with them being 10x only is that backports would potentially be more difficult for all the life of 9.
>>> >>>
>>> >>> On Mon, Oct 5, 2020 at 4:54 PM Noble Paul <[hidden email]> wrote:
>>> >>>>
>>> >>>> >I don't think it can be said what committers do and don't do with regards to running Solr.  All of us would answer this differently and at different points in time.
>>> >>>>
>>> >>>> " I have run it in one large cluster, so it is certified to be bug free/stable" I don't think it's a reasonable approach. We need as much feedback from our users because each of them stress Solr in a different way. This is not to suggest that committers are not doing testing or their tests are not valid. When I talk to the committers out here they say they do not see any performance stability issues at all. But, my client reports issues on a day to day basis.
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> > Definitely publish a Docker image BTW -- it's the best way to try out any software.
>>> >>>>
>>> >>>> Docker is not a big requirement for large scale installations. Most of them already have their own install scripts. Availability of docker is not important for them. If a user is only encouraged to install Solr because of a docker image , most likely they are not running a large enough cluster
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> On Tue, Oct 6, 2020, 6:30 AM David Smiley <[hidden email]> wrote:
>>> >>>>>
>>> >>>>> Thanks so much for your responses Ishan... I'm getting much more information in this thread than my attempts to get questions answered on the JIRA issue months ago.  And especially,  thank you for volunteering for the difficult porting efforts!
>>> >>>>>
>>> >>>>> Tomas said:
>>> >>>>>>
>>> >>>>>>  I do agree with the previous comments that calling it "Solr 10" (even with the "-alpha") would confuse users, maybe use "reference"? or maybe something in reference to SOLR-14788?
>>> >>>>>
>>> >>>>>
>>> >>>>> I have the opposite opinion.  This word "reference" is baffling to me despite whatever Mark's explanation is.  I like the justification Ishan gave for 10-alpha and I don't think I could re-phrase his justification any better.  *If* the release was _not_ official (thus wouldn't show up in the usual places anyone would look for a release), I think it would alleviate that confusion concern even more, although I think "alpha" ought to be enough of a signal not to use it without digging deeper on what's going on.
>>> >>>>>
>>> >>>>> Alex then Ishan said:
>>> >>>>>>
>>> >>>>>> > Maybe we could release it to
>>> >>>>>> > committers community first and dogfood it "internally"?
>>> >>>>>>
>>> >>>>>> Alex: It is meaningless. Committers don't run large scale installations. We barely even have time to take care of running unit tests before destabilizing our builds. We are not the right audience. However, we all can anyway check out the branch and start playing with it, even without a release. There are orgs that don't want to install any code that wasn't officially released; this release is geared towards them (to help us test this at their scale).
>>> >>>>>
>>> >>>>>
>>> >>>>> I don't think it can be said what committers do and don't do with regards to running Solr.  All of us would answer this differently and at different points in time.  From time to time, though not at present, I've been well positioned to try out a new version of Solr in a stage/test environment to see how it goes.  (Putting on my Salesforce metaphorical hat...) Even though I'm not able to deploy it in a realistic way today, I'm able to run a battery of tests to see if one of the features we depend on have changed or is broken.  That's useful feedback to an alpha release!  And even though I'm saying I'm not well positioned to try out some new Solr release in a production-ish setting now, it's something I could make a good case for internally since upgrades take a lot of effort where I work.  It's in our interest for SolrCloud to be very stable (of course).
>>> >>>>>
>>> >>>>> Regardless, I think what you're driving at Ishan is that you want an "official" release -- one that goes through the whole ceremony.  You believe that people would be more likely to use it.  I think all we need to do is announce (similar to a real release) that there is some unofficial alpha distribution and that we want to solicit your feedback -- basically, help us find bugs.  Definitely publish a Docker image BTW -- it's the best way to try out any software.  I'm -0 on doing an official release for alpha software because it's unnecessary to achieve the goals and somewhat confusing.  I think the Solr 4 alpha/beta situation was different -- it was not some fork a committer was maintaining; it was the master branch of its time, and it was destined to be the very next release, not some possible future release.
>>> >>>>>
>>> >>>>> ~ David Smiley
>>> >>>>> Apache Lucene/Solr Search Developer
>>> >>>>> http://www.linkedin.com/in/davidwsmiley
>>>
>>>
>>>
>>> --
>>> -----------------------------------------------------
>>> Noble Paul
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: [hidden email]
>>> For additional commands, e-mail: [hidden email]
>>>


--
-----------------------------------------------------
Noble Paul

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Solr Alpha (EA) release of Reference Branch

Ilan Ginzburg
Copying below Mark's posts from ASF Slack #solr-next-big-thing channel.

The Solr Reference Branch.
Document 1, a quick intro.
You can think of the Solr Reference Branch as a remaster of Solr. It
is not an attempt to redesign Solr or make it more fancy. The goal of
the Solr Reference Branch is to be a better incarnation of the current
Apache Solr, which will provide a base for future development and
design.
There are a variety of problems with Solr today that make it difficult
to adopt and run. This is me being as honest and objective as I can
be, though no doubt, many will see it as an exaggeration or negative
focus. I just see it as the way it is and has been, it's just taken me
a real long time to actually get all the way under the rug to find the
really hardened nasty cockroaches burrowed in there.
1. Resource usage and management is wasteful, inefficient, buggy, and haphazard.
2. SolrCloud is not long term reliable. Exceptional cases will
frequently flummox the system, and exceptional cases are supposed to
be our wheelhouse and primary focus. Leaders will be lost and not
recover, the Overseer will go away, GC storms will hit, tight loops in
a bad case will crank up resources, and retries will be abundant and
overaggressive.
3. Our blocking and locking is generally not efficient, especially in key paths.
4. We get thread safety wrong (too often) in some important spots.
5. Distributed updates have to be added locally before they are
distributed, and then that distribution is generally inefficient,
prone to blocking and/or timeouts, and hobbled by HTTP1.1 and our need
for packing updates into a single request to achieve any kind of
performance, losing proper error handling and eating the many rough
edges of the ConcurrentUpdateSolrClient.
6. Our Zookeeper foundation code is often inefficient, buggy,
unreliable, and improperly used (we don’t always use async or multi
where we should, we force updates from zk instead of being notified,
we don’t handle session expiration as well as we should, our
algorithms are slow and buggy, we make a multitude more calls than we
should (especially on cluster startup), etc, etc)
7. We have circular dependencies between major classes that can start
threads in their constructors that start interacting with the other
classes before construction is complete.
8. Our XML handling is abysmally outdated and slow for multiple
reasons. Our heavy Xpath usage is incredibly wasteful and expensive.
9. Our thread management is not understandable, not properly tunable,
not efficient, sometimes buggy, not always consistent, and difficult
to understand fundamentally.
10. Our Jetty configuration is lacking in a variety of ways,
especially around shutdown and http2.
11. The dynamic schema feature can be very expensive and not fully thread safe.
12. The Overseer is extremely inefficient, can be extremely slow to
stop, had a buggy leader election algorithm, doesn’t handle session
expiration as well as it should, can keep trying to come back from the
dead, and the list goes on.
13. Our connection resuse is often very poor or non existent, when
it’s improved, it always reverts back to bad or worse.
14. HTTP1.1 is not great for our type of application in a variety of
ways that HTTP2 solves – but we still use a lot of HTTP1.1 and HTTP2
is not configured well and the client needs some work.
15. Lifecycle of important objects is often off, most things can and
will leak (SolrCores, SolrIndexSearchers, Directory’s, Solr clients),
some things will close objects more than once or that don’t belong to
them, or close things in a bad order.
16. There is often sleeps and/or pulling that is a magnitude slower
than proper event driven waits.
17. Our tests are actually pretty unstable and making them stable is
way, way more difficult than most people realize. I’m quite sure I’ve
spent much, much more time on this than anyone out there, and I can
tell you, the tests are not stable in a 1,000 shifting ways that have
and will continue to cause lots of damage.
18. We don’t have good async update/search support for scaling and
better resource usage.
19. We often duplicate resources or create new pools instead of sharing.
20. We don’t do tons of parallelizable stuff in parallel, when we do
it’s inconsistent.
21. Our Collections API can often not wait correctly for the proper
state for what it did to be ready before returning. Even if it gets it
right, a cloud client that made the request won’t necessarily have the
updated state local when the request returns. Things often still work,
but with a variety of interesting and slow results possible.
22. We don’t often holistically look at what we have built and how it
fits together and so often there are silly things, bad fits, one off
bad patterns, lazy attempts at something, etc.
24. Close and shutdown are inefficient and slow across a huge swatch
of our object tree. These issues tend to be growy and breed less
concern over time.
25. There are a variety of ways and places that we can generate an
absurd amount of unnecessary garbage.
26. SolrCore reload is not fully reliable and increasingly important and used.
27. The leader election code has a variety of ugly little bugs and is
based on a recursive implementation that will eventually exhaust stack
space – though it’s likely your cluster will be brought down by
something else before that is a problem (unless you hit the infinite
loop, no one can be leader, eat up the stack as fast as possible case
– which should be hard these days with the leader election throttle).
28. The recovery processes, like almost everything you can imagine,
has a variety of issues and rarer bad cases and affects.
By and large, everything is inefficient and buggy and full of accepted
compromise regardless.
Interestingly, this does not make us an atypical open source Java
distributed project. But, I’m kind of a software snob, and I would not
run this thing and so I cannot work on it. What is there to do ...
The Solr Reference Branch is intended to tackle every one of those
issues. As well as about 1000+ more of varying and lesser importance.
As all of that comes together, cool stuff starts to unlock, and you
begin to see some phenomena that is together much greater than the sum
of it’s many, many parts.
29. Our tests have been getting better and better are stamping out the
legit noise they create - every scream a breadcrumb towards badness -
but we have built a scream catching machine - though we will never be
able to catch them all for a huge variety of deep reasons.

The Solr Reference Branch
Document 2
While the extent of the previously mentioned issues was not clear to
me, that is a deep rabbit hole, I’ve always, as have many others,
known the current state of things with Solr at a higher, broader
level.
So what about this effort is different? Is this not just a bunch of my
standard JIRA issues all crammed into one? Should we not break them
out proper and do things sensible?
Well, previously, as is probably common, I was both a bit lost on
where we were exactly and certainly on where to find firmer ground for
real, not just the mirage always just over the hill.
I love performance and efficiency though. I’ve always avoided it as a
focus with Solr and SolrCloud, thinking stability has to come first.
Having given up on stability and scale after a good 8 years or
something, completely tossed out as a pipe dream, I started work on
something new, something really just for me. I started plugging in
HTTP2. And the effort and work needed for that and the learning and
some of the results, completely opened my eyes. I also attacked very
different than I have in the past, doing something I like for me, I
drowned myself in it. Spent 2-3 weeks at a time here and there sitting
at the computer with intense focus for 16-20 hours a day. The more I
did, the more I found, the more I understood, the more I discovered.
I discovered a discovery processes. It was leading me to everything I
needed to do and I just had to follow the long, ever flowing path,
keeping my mental models strong, re etching, ruminating, obsessing.
I realized many test functions we have – most- should be taking on the
order of milliseconds instead of seconds to dozens of seconds. I
realized tons and tons of our issues and gremlins lived and prospered
in our slow and inefficient smog. I realized that if I just spent the
time to look where slowness and flakiness prevailed, really look, like
take hours just for some random side road - build a bridge, burn it,
and build one further down, etc, etc – that making huge improvement
after huge improvement was actually very low hanging fruit, just
hidden by some thorns and leaves and lack of any reasonable
introspection into the system we have created and continue to build.
Over time, I could see what had to be done and I could see what it
would achieve. I built different parts at different times, lost them
and rebuilt them a different way with different focus. I build and
expanded my introspection and measuring tools and classes.
That’s a sentence trying to cover a universe, but if you want to
really boil it down even further, I’d invoke the normally faulty
broken windows theory. There is magic in perfect windows that only
those that have them know. Can we get perfect? I like to dream and
there is no end to the introspection, experimentation, and
improvements to try. The perfect landing aside though, no doubt we can
move drastically from where we are.

Another thing I learned is the crazy number of ways you can make all
the tests pass like champions, and roll into production unusable.
Which tells me that production users are a large part of our test
strategy, and that can’t be to make any real change in a satisfactory
way.

The current goal is to have a mostly usable and testable system by
mid-late October. Not everything 100%, some known caveats and cleanup
and plenty to do, but it should be in good shape for a user to try out
given the caveats outlined
The biggest risk currently is the absorption of the search side async
work from master - I'm familiar with that, I've worked on it myself,
the code involved is derived from an old branch of mine, but async is
a whole different animal and trying to nail it without any downsides
to the old synchronous model is a tough nut
one that I was already battling on the dist update side, so it's good
stuff to work on and do, but its taking some effort to get in shape

On Tue, Oct 6, 2020 at 8:00 PM Tomás Fernández Löbbe
<[hidden email]> wrote:

>
> > Let's say we cut 9x and now there is a new master taken from the reference branch.
> I never said “make a new master”, I said merge changes in ref branch into master. If things are broken into pieces like Ishan is suggesting, those changes can be merged into 9.x too. I only suggested this because you felt unsure about merging to master now and I guess this is due to fear of introducing bugs so close to a potential 9.0 release, is that not right?
>
>
> > We will never be able to reconcile these 2 branches
> Sorry, but how is that different if we do an alpha release from the branch now? What would be the process after that? Let's say people don't find issues and we want to merge those changes, what’s the plan then?
>
> > Choice 1:
> I’m fine with choice 1 if that’s what you want, as long as it’s not an official release for the reasons stated above.
>
>
> > I promise to do code review & cleanup as much as possible. But I'm hesitant to give a stamp of approval to make it THE official release
> What do you mean? I thought this is what you were suggesting, make an official release from the reference_impl branch?
>
>
> I think Ilan’s last email is on spot, and I agree 100% with what he can express much better than I can :)
>
> > Mark's descriptions in Slack go in the right way but are still too high level
> Can someone share those here? or in Jira?
>
> On Tue, Oct 6, 2020 at 5:09 AM Noble Paul <[hidden email]> wrote:
>>
>> > I think the danger is high to treat this branch as a black box (or an "all or nothing").
>>
>> True Ilan.  Ideally, I would like a few of us to study the code &
>> start pulling in changes we are confident of (even to 8x branch, why
>> not). We cannot burden a single developer to do everything.
>>
>> This cannot be a task just for one or 2 devs. We all will have to work
>> together to decompose the changes and digest them into master. I can
>> do my bit.
>>
>> But, I'm sure we may hit a point where certain changes cannot be
>> isolated and absorbed. We will have to collectively make a call, how
>> to absorb them
>>
>> On Tue, Oct 6, 2020 at 9:00 PM Ishan Chattopadhyaya
>> <[hidden email]> wrote:
>> >
>> >
>> > I'm willing to help and I believe others will too if the amount of work for contributing is reasonable (i.e. not a three months effort).
>> >
>> > I looked into the possibility of doing so. To me, it seemed to be that it is very hard to do so: possibly 1 year project for me. Problem is that it is hard to pull out a particular class of improvements (say thread management improvement) and have all tests pass with it (because tests have gotten extensive improvements of their own) and also observe the effect of the improvement. IIUC, every improvement to Solr seemed to require many iterations to get the tests happy. I remember Mark telling me that it may not even be possible for him to do something like that (i.e. bring all changes into master as tiny pieces).
>> >
>> > What I volunteered to do, however, is to decompose roughly all the general improvements into smaller, manageable commits. However, making sure all tests pass at every commit point is beyond my capability.
>> >
>> > On Tue, 6 Oct, 2020, 3:10 pm Ilan Ginzburg, <[hidden email]> wrote:
>> >>
>> >> Another option to integrate this work into the main code line would be to understand what changes have been made and where (Mark's descriptions in Slack go in the right way but are still too high level), and then port or even redo them in main, one by one.
>> >>
>> >> I think the danger is high to treat this branch as a black box (or an "all or nothing"). Using the merging itself to change our understanding and increase our knowledge of what was done can greatly reduce the risk.
>> >>
>> >> We do develop new features in Solr 9 without beta releasing them, so if we port Mark's improvements by small chunks (and maybe in the process decide that some should not be ported or not now) I don't see why this can't integrate to become like other improvements done to the code. If specific changes do require a beta release, do that release from master and pick the right moment.
>> >>
>> >> I'm willing to help and I believe others will too if the amount of work for contributing is reasonable (i.e. not a three months effort). This requires documenting the changes done in that branch, pointing to where these changes happened and then picking them up one by one and porting them more or less independently of each other. We might only port a subset of changes by the time 9.0 is released, that's fine we can continue in following releases.
>> >>
>> >> My 2 cents...
>> >> Ilan
>> >>
>> >> Le mar. 6 oct. 2020 à 09:56, Noble Paul <[hidden email]> a écrit :
>> >>>
>> >>> Yes, A docker image will definitely help. I wasn't trying to downplay that
>> >>>
>> >>> On Tue, Oct 6, 2020 at 6:55 PM Ishan Chattopadhyaya
>> >>> <[hidden email]> wrote:
>> >>> >
>> >>> >
>> >>> > > Docker is not a big requirement for large scale installations. Most of them already have their own install scripts. Availability of docker is not important for them. If a user is only encouraged to install Solr because of a docker image , most likely they are not running a large enough cluster
>> >>> >
>> >>> > I disagree, Noble. Having a docker image us going to be useful to some clients, with complex usecases. Great point, David!
>> >>> >
>> >>> > On Tue, 6 Oct, 2020, 1:09 pm Ishan Chattopadhyaya, <[hidden email]> wrote:
>> >>> >>
>> >>> >> As I said, I'm *personally* not confident in putting such a big changeset into master that wasn't vetted in a real user environment widely. I have, in the past, done enough bad things to Solr (directly or indirectly), and I don't want to repeat the same. Also, I'll be very uncomfortable if someone else did so.
>> >>> >>
>> >>> >> Having said this, if someone else wants to port the changes over to master *without first getting enough real world testing*, feel free to do so, and I can focus my efforts elsewhere.
>> >>> >>
>> >>> >> On Tue, 6 Oct, 2020, 9:22 am Tomás Fernández Löbbe, <[hidden email]> wrote:
>> >>> >>>
>> >>> >>> I was thinking (and I haven’t flushed it out completely but will throw the idea) that an alternative approach with this timeline could be to cut 9x branch around November/December? And then you could merge into master, it would have the latest  changes from master plus the ref branch changes. From there any nightly build could be use to help test/debug.
>> >>> >>>
>> >>> >>> That said I don’t know for sure what are the changes in the branch that do not belong in 9. The problem with them being 10x only is that backports would potentially be more difficult for all the life of 9.
>> >>> >>>
>> >>> >>> On Mon, Oct 5, 2020 at 4:54 PM Noble Paul <[hidden email]> wrote:
>> >>> >>>>
>> >>> >>>> >I don't think it can be said what committers do and don't do with regards to running Solr.  All of us would answer this differently and at different points in time.
>> >>> >>>>
>> >>> >>>> " I have run it in one large cluster, so it is certified to be bug free/stable" I don't think it's a reasonable approach. We need as much feedback from our users because each of them stress Solr in a different way. This is not to suggest that committers are not doing testing or their tests are not valid. When I talk to the committers out here they say they do not see any performance stability issues at all. But, my client reports issues on a day to day basis.
>> >>> >>>>
>> >>> >>>>
>> >>> >>>>
>> >>> >>>> > Definitely publish a Docker image BTW -- it's the best way to try out any software.
>> >>> >>>>
>> >>> >>>> Docker is not a big requirement for large scale installations. Most of them already have their own install scripts. Availability of docker is not important for them. If a user is only encouraged to install Solr because of a docker image , most likely they are not running a large enough cluster
>> >>> >>>>
>> >>> >>>>
>> >>> >>>>
>> >>> >>>> On Tue, Oct 6, 2020, 6:30 AM David Smiley <[hidden email]> wrote:
>> >>> >>>>>
>> >>> >>>>> Thanks so much for your responses Ishan... I'm getting much more information in this thread than my attempts to get questions answered on the JIRA issue months ago.  And especially,  thank you for volunteering for the difficult porting efforts!
>> >>> >>>>>
>> >>> >>>>> Tomas said:
>> >>> >>>>>>
>> >>> >>>>>>  I do agree with the previous comments that calling it "Solr 10" (even with the "-alpha") would confuse users, maybe use "reference"? or maybe something in reference to SOLR-14788?
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> I have the opposite opinion.  This word "reference" is baffling to me despite whatever Mark's explanation is.  I like the justification Ishan gave for 10-alpha and I don't think I could re-phrase his justification any better.  *If* the release was _not_ official (thus wouldn't show up in the usual places anyone would look for a release), I think it would alleviate that confusion concern even more, although I think "alpha" ought to be enough of a signal not to use it without digging deeper on what's going on.
>> >>> >>>>>
>> >>> >>>>> Alex then Ishan said:
>> >>> >>>>>>
>> >>> >>>>>> > Maybe we could release it to
>> >>> >>>>>> > committers community first and dogfood it "internally"?
>> >>> >>>>>>
>> >>> >>>>>> Alex: It is meaningless. Committers don't run large scale installations. We barely even have time to take care of running unit tests before destabilizing our builds. We are not the right audience. However, we all can anyway check out the branch and start playing with it, even without a release. There are orgs that don't want to install any code that wasn't officially released; this release is geared towards them (to help us test this at their scale).
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> I don't think it can be said what committers do and don't do with regards to running Solr.  All of us would answer this differently and at different points in time.  From time to time, though not at present, I've been well positioned to try out a new version of Solr in a stage/test environment to see how it goes.  (Putting on my Salesforce metaphorical hat...) Even though I'm not able to deploy it in a realistic way today, I'm able to run a battery of tests to see if one of the features we depend on have changed or is broken.  That's useful feedback to an alpha release!  And even though I'm saying I'm not well positioned to try out some new Solr release in a production-ish setting now, it's something I could make a good case for internally since upgrades take a lot of effort where I work.  It's in our interest for SolrCloud to be very stable (of course).
>> >>> >>>>>
>> >>> >>>>> Regardless, I think what you're driving at Ishan is that you want an "official" release -- one that goes through the whole ceremony.  You believe that people would be more likely to use it.  I think all we need to do is announce (similar to a real release) that there is some unofficial alpha distribution and that we want to solicit your feedback -- basically, help us find bugs.  Definitely publish a Docker image BTW -- it's the best way to try out any software.  I'm -0 on doing an official release for alpha software because it's unnecessary to achieve the goals and somewhat confusing.  I think the Solr 4 alpha/beta situation was different -- it was not some fork a committer was maintaining; it was the master branch of its time, and it was destined to be the very next release, not some possible future release.
>> >>> >>>>>
>> >>> >>>>> ~ David Smiley
>> >>> >>>>> Apache Lucene/Solr Search Developer
>> >>> >>>>> http://www.linkedin.com/in/davidwsmiley
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> -----------------------------------------------------
>> >>> Noble Paul
>> >>>
>> >>> ---------------------------------------------------------------------
>> >>> To unsubscribe, e-mail: [hidden email]
>> >>> For additional commands, e-mail: [hidden email]
>> >>>
>>
>>
>> --
>> -----------------------------------------------------
>> Noble Paul
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Solr Alpha (EA) release of Reference Branch

Anshum Gupta
Thanks for initiating this discussion, Ishan. 

For the sake of making sure that we are all on the same page, let me summarize my understanding and take on this thread.

The current situation
Mark has a reference branch, which the folks who have looked at the branch, feel that it’s a much better, improved, reliable, and sustainable version of the current master i.e. take the same baseline and make it better. We would like to get those changes to the project, but aren’t sure about how to do so. Releasing the branch when it’s ready to go, as an alpha release will allow users to test it.

1. Is releasing the branch officially going to help us achieve the goal of having a well tested branch ?
2. Assuming #1 is true, do we as a community want to release the branch officially and assume responsibility?
3. What is our path forward after the release I.e. do we merge the branch into master or swap out current master.

What do we plan to do (options).
I feel there is a consensus on everyone wanting the best for the project and wanting Marks’ changes released. 

#1 - There are differing opinions, and I personally think we can have our test harnesses test the new branch, but I think most companies running Solr at scale would have concerns with taking up an alpha release and deploying it in production. The various tests that a bunch of folks are working on is our best bet at testing out the branch, in which case I’m not sure if we want an official release.

#2 - I feel that having an official release and having artifacts show up in maven central will confuse people. The 4.0 alpha release was very different in the sense that it was the same branch, the code wasn’t replacing anything existing but introducing a completely new feature i.e. SolrCloud.

#3 - I’m still unclear on how these changes will be released in terms of the community consensus. I’ve tried to merge parts of Marks’ effort from another time into master, but it’s very difficult, almost impossible to isolate and extract commits on the basis of coverage/features/etc. This is a lot of really great effort and after having spoken with Mark multiple times, I really feel we should figure out a way to absorb this but I do have concerns around replacing the master branch completely.

While I do like the idea that Tomás proposed, I also feel that maintaining and managing cherry-picking across 9x, master, and ref branch will only make it difficult for people to work though the duration of 9x. 

I haven’t looked at the current ref branch recently, but the folks who have looked at it, if you think that this code can be merged into master even as big chunks, that’d be the most confidence building way forward. 




On Tue, Oct 6, 2020 at 11:37 AM Ilan Ginzburg <[hidden email]> wrote:
Copying below Mark's posts from ASF Slack #solr-next-big-thing channel.

The Solr Reference Branch.
Document 1, a quick intro.
You can think of the Solr Reference Branch as a remaster of Solr. It
is not an attempt to redesign Solr or make it more fancy. The goal of
the Solr Reference Branch is to be a better incarnation of the current
Apache Solr, which will provide a base for future development and
design.
There are a variety of problems with Solr today that make it difficult
to adopt and run. This is me being as honest and objective as I can
be, though no doubt, many will see it as an exaggeration or negative
focus. I just see it as the way it is and has been, it's just taken me
a real long time to actually get all the way under the rug to find the
really hardened nasty cockroaches burrowed in there.
1. Resource usage and management is wasteful, inefficient, buggy, and haphazard.
2. SolrCloud is not long term reliable. Exceptional cases will
frequently flummox the system, and exceptional cases are supposed to
be our wheelhouse and primary focus. Leaders will be lost and not
recover, the Overseer will go away, GC storms will hit, tight loops in
a bad case will crank up resources, and retries will be abundant and
overaggressive.
3. Our blocking and locking is generally not efficient, especially in key paths.
4. We get thread safety wrong (too often) in some important spots.
5. Distributed updates have to be added locally before they are
distributed, and then that distribution is generally inefficient,
prone to blocking and/or timeouts, and hobbled by HTTP1.1 and our need
for packing updates into a single request to achieve any kind of
performance, losing proper error handling and eating the many rough
edges of the ConcurrentUpdateSolrClient.
6. Our Zookeeper foundation code is often inefficient, buggy,
unreliable, and improperly used (we don’t always use async or multi
where we should, we force updates from zk instead of being notified,
we don’t handle session expiration as well as we should, our
algorithms are slow and buggy, we make a multitude more calls than we
should (especially on cluster startup), etc, etc)
7. We have circular dependencies between major classes that can start
threads in their constructors that start interacting with the other
classes before construction is complete.
8. Our XML handling is abysmally outdated and slow for multiple
reasons. Our heavy Xpath usage is incredibly wasteful and expensive.
9. Our thread management is not understandable, not properly tunable,
not efficient, sometimes buggy, not always consistent, and difficult
to understand fundamentally.
10. Our Jetty configuration is lacking in a variety of ways,
especially around shutdown and http2.
11. The dynamic schema feature can be very expensive and not fully thread safe.
12. The Overseer is extremely inefficient, can be extremely slow to
stop, had a buggy leader election algorithm, doesn’t handle session
expiration as well as it should, can keep trying to come back from the
dead, and the list goes on.
13. Our connection resuse is often very poor or non existent, when
it’s improved, it always reverts back to bad or worse.
14. HTTP1.1 is not great for our type of application in a variety of
ways that HTTP2 solves – but we still use a lot of HTTP1.1 and HTTP2
is not configured well and the client needs some work.
15. Lifecycle of important objects is often off, most things can and
will leak (SolrCores, SolrIndexSearchers, Directory’s, Solr clients),
some things will close objects more than once or that don’t belong to
them, or close things in a bad order.
16. There is often sleeps and/or pulling that is a magnitude slower
than proper event driven waits.
17. Our tests are actually pretty unstable and making them stable is
way, way more difficult than most people realize. I’m quite sure I’ve
spent much, much more time on this than anyone out there, and I can
tell you, the tests are not stable in a 1,000 shifting ways that have
and will continue to cause lots of damage.
18. We don’t have good async update/search support for scaling and
better resource usage.
19. We often duplicate resources or create new pools instead of sharing.
20. We don’t do tons of parallelizable stuff in parallel, when we do
it’s inconsistent.
21. Our Collections API can often not wait correctly for the proper
state for what it did to be ready before returning. Even if it gets it
right, a cloud client that made the request won’t necessarily have the
updated state local when the request returns. Things often still work,
but with a variety of interesting and slow results possible.
22. We don’t often holistically look at what we have built and how it
fits together and so often there are silly things, bad fits, one off
bad patterns, lazy attempts at something, etc.
24. Close and shutdown are inefficient and slow across a huge swatch
of our object tree. These issues tend to be growy and breed less
concern over time.
25. There are a variety of ways and places that we can generate an
absurd amount of unnecessary garbage.
26. SolrCore reload is not fully reliable and increasingly important and used.
27. The leader election code has a variety of ugly little bugs and is
based on a recursive implementation that will eventually exhaust stack
space – though it’s likely your cluster will be brought down by
something else before that is a problem (unless you hit the infinite
loop, no one can be leader, eat up the stack as fast as possible case
– which should be hard these days with the leader election throttle).
28. The recovery processes, like almost everything you can imagine,
has a variety of issues and rarer bad cases and affects.
By and large, everything is inefficient and buggy and full of accepted
compromise regardless.
Interestingly, this does not make us an atypical open source Java
distributed project. But, I’m kind of a software snob, and I would not
run this thing and so I cannot work on it. What is there to do ...
The Solr Reference Branch is intended to tackle every one of those
issues. As well as about 1000+ more of varying and lesser importance.
As all of that comes together, cool stuff starts to unlock, and you
begin to see some phenomena that is together much greater than the sum
of it’s many, many parts.
29. Our tests have been getting better and better are stamping out the
legit noise they create - every scream a breadcrumb towards badness -
but we have built a scream catching machine - though we will never be
able to catch them all for a huge variety of deep reasons.

The Solr Reference Branch
Document 2
While the extent of the previously mentioned issues was not clear to
me, that is a deep rabbit hole, I’ve always, as have many others,
known the current state of things with Solr at a higher, broader
level.
So what about this effort is different? Is this not just a bunch of my
standard JIRA issues all crammed into one? Should we not break them
out proper and do things sensible?
Well, previously, as is probably common, I was both a bit lost on
where we were exactly and certainly on where to find firmer ground for
real, not just the mirage always just over the hill.
I love performance and efficiency though. I’ve always avoided it as a
focus with Solr and SolrCloud, thinking stability has to come first.
Having given up on stability and scale after a good 8 years or
something, completely tossed out as a pipe dream, I started work on
something new, something really just for me. I started plugging in
HTTP2. And the effort and work needed for that and the learning and
some of the results, completely opened my eyes. I also attacked very
different than I have in the past, doing something I like for me, I
drowned myself in it. Spent 2-3 weeks at a time here and there sitting
at the computer with intense focus for 16-20 hours a day. The more I
did, the more I found, the more I understood, the more I discovered.
I discovered a discovery processes. It was leading me to everything I
needed to do and I just had to follow the long, ever flowing path,
keeping my mental models strong, re etching, ruminating, obsessing.
I realized many test functions we have – most- should be taking on the
order of milliseconds instead of seconds to dozens of seconds. I
realized tons and tons of our issues and gremlins lived and prospered
in our slow and inefficient smog. I realized that if I just spent the
time to look where slowness and flakiness prevailed, really look, like
take hours just for some random side road - build a bridge, burn it,
and build one further down, etc, etc – that making huge improvement
after huge improvement was actually very low hanging fruit, just
hidden by some thorns and leaves and lack of any reasonable
introspection into the system we have created and continue to build.
Over time, I could see what had to be done and I could see what it
would achieve. I built different parts at different times, lost them
and rebuilt them a different way with different focus. I build and
expanded my introspection and measuring tools and classes.
That’s a sentence trying to cover a universe, but if you want to
really boil it down even further, I’d invoke the normally faulty
broken windows theory. There is magic in perfect windows that only
those that have them know. Can we get perfect? I like to dream and
there is no end to the introspection, experimentation, and
improvements to try. The perfect landing aside though, no doubt we can
move drastically from where we are.

Another thing I learned is the crazy number of ways you can make all
the tests pass like champions, and roll into production unusable.
Which tells me that production users are a large part of our test
strategy, and that can’t be to make any real change in a satisfactory
way.

The current goal is to have a mostly usable and testable system by
mid-late October. Not everything 100%, some known caveats and cleanup
and plenty to do, but it should be in good shape for a user to try out
given the caveats outlined
The biggest risk currently is the absorption of the search side async
work from master - I'm familiar with that, I've worked on it myself,
the code involved is derived from an old branch of mine, but async is
a whole different animal and trying to nail it without any downsides
to the old synchronous model is a tough nut
one that I was already battling on the dist update side, so it's good
stuff to work on and do, but its taking some effort to get in shape

On Tue, Oct 6, 2020 at 8:00 PM Tomás Fernández Löbbe
<[hidden email]> wrote:
>
> > Let's say we cut 9x and now there is a new master taken from the reference branch.
> I never said “make a new master”, I said merge changes in ref branch into master. If things are broken into pieces like Ishan is suggesting, those changes can be merged into 9.x too. I only suggested this because you felt unsure about merging to master now and I guess this is due to fear of introducing bugs so close to a potential 9.0 release, is that not right?
>
>
> > We will never be able to reconcile these 2 branches
> Sorry, but how is that different if we do an alpha release from the branch now? What would be the process after that? Let's say people don't find issues and we want to merge those changes, what’s the plan then?
>
> > Choice 1:
> I’m fine with choice 1 if that’s what you want, as long as it’s not an official release for the reasons stated above.
>
>
> > I promise to do code review & cleanup as much as possible. But I'm hesitant to give a stamp of approval to make it THE official release
> What do you mean? I thought this is what you were suggesting, make an official release from the reference_impl branch?
>
>
> I think Ilan’s last email is on spot, and I agree 100% with what he can express much better than I can :)
>
> > Mark's descriptions in Slack go in the right way but are still too high level
> Can someone share those here? or in Jira?
>
> On Tue, Oct 6, 2020 at 5:09 AM Noble Paul <[hidden email]> wrote:
>>
>> > I think the danger is high to treat this branch as a black box (or an "all or nothing").
>>
>> True Ilan.  Ideally, I would like a few of us to study the code &
>> start pulling in changes we are confident of (even to 8x branch, why
>> not). We cannot burden a single developer to do everything.
>>
>> This cannot be a task just for one or 2 devs. We all will have to work
>> together to decompose the changes and digest them into master. I can
>> do my bit.
>>
>> But, I'm sure we may hit a point where certain changes cannot be
>> isolated and absorbed. We will have to collectively make a call, how
>> to absorb them
>>
>> On Tue, Oct 6, 2020 at 9:00 PM Ishan Chattopadhyaya
>> <[hidden email]> wrote:
>> >
>> >
>> > I'm willing to help and I believe others will too if the amount of work for contributing is reasonable (i.e. not a three months effort).
>> >
>> > I looked into the possibility of doing so. To me, it seemed to be that it is very hard to do so: possibly 1 year project for me. Problem is that it is hard to pull out a particular class of improvements (say thread management improvement) and have all tests pass with it (because tests have gotten extensive improvements of their own) and also observe the effect of the improvement. IIUC, every improvement to Solr seemed to require many iterations to get the tests happy. I remember Mark telling me that it may not even be possible for him to do something like that (i.e. bring all changes into master as tiny pieces).
>> >
>> > What I volunteered to do, however, is to decompose roughly all the general improvements into smaller, manageable commits. However, making sure all tests pass at every commit point is beyond my capability.
>> >
>> > On Tue, 6 Oct, 2020, 3:10 pm Ilan Ginzburg, <[hidden email]> wrote:
>> >>
>> >> Another option to integrate this work into the main code line would be to understand what changes have been made and where (Mark's descriptions in Slack go in the right way but are still too high level), and then port or even redo them in main, one by one.
>> >>
>> >> I think the danger is high to treat this branch as a black box (or an "all or nothing"). Using the merging itself to change our understanding and increase our knowledge of what was done can greatly reduce the risk.
>> >>
>> >> We do develop new features in Solr 9 without beta releasing them, so if we port Mark's improvements by small chunks (and maybe in the process decide that some should not be ported or not now) I don't see why this can't integrate to become like other improvements done to the code. If specific changes do require a beta release, do that release from master and pick the right moment.
>> >>
>> >> I'm willing to help and I believe others will too if the amount of work for contributing is reasonable (i.e. not a three months effort). This requires documenting the changes done in that branch, pointing to where these changes happened and then picking them up one by one and porting them more or less independently of each other. We might only port a subset of changes by the time 9.0 is released, that's fine we can continue in following releases.
>> >>
>> >> My 2 cents...
>> >> Ilan
>> >>
>> >> Le mar. 6 oct. 2020 à 09:56, Noble Paul <[hidden email]> a écrit :
>> >>>
>> >>> Yes, A docker image will definitely help. I wasn't trying to downplay that
>> >>>
>> >>> On Tue, Oct 6, 2020 at 6:55 PM Ishan Chattopadhyaya
>> >>> <[hidden email]> wrote:
>> >>> >
>> >>> >
>> >>> > > Docker is not a big requirement for large scale installations. Most of them already have their own install scripts. Availability of docker is not important for them. If a user is only encouraged to install Solr because of a docker image , most likely they are not running a large enough cluster
>> >>> >
>> >>> > I disagree, Noble. Having a docker image us going to be useful to some clients, with complex usecases. Great point, David!
>> >>> >
>> >>> > On Tue, 6 Oct, 2020, 1:09 pm Ishan Chattopadhyaya, <[hidden email]> wrote:
>> >>> >>
>> >>> >> As I said, I'm *personally* not confident in putting such a big changeset into master that wasn't vetted in a real user environment widely. I have, in the past, done enough bad things to Solr (directly or indirectly), and I don't want to repeat the same. Also, I'll be very uncomfortable if someone else did so.
>> >>> >>
>> >>> >> Having said this, if someone else wants to port the changes over to master *without first getting enough real world testing*, feel free to do so, and I can focus my efforts elsewhere.
>> >>> >>
>> >>> >> On Tue, 6 Oct, 2020, 9:22 am Tomás Fernández Löbbe, <[hidden email]> wrote:
>> >>> >>>
>> >>> >>> I was thinking (and I haven’t flushed it out completely but will throw the idea) that an alternative approach with this timeline could be to cut 9x branch around November/December? And then you could merge into master, it would have the latest  changes from master plus the ref branch changes. From there any nightly build could be use to help test/debug.
>> >>> >>>
>> >>> >>> That said I don’t know for sure what are the changes in the branch that do not belong in 9. The problem with them being 10x only is that backports would potentially be more difficult for all the life of 9.
>> >>> >>>
>> >>> >>> On Mon, Oct 5, 2020 at 4:54 PM Noble Paul <[hidden email]> wrote:
>> >>> >>>>
>> >>> >>>> >I don't think it can be said what committers do and don't do with regards to running Solr.  All of us would answer this differently and at different points in time.
>> >>> >>>>
>> >>> >>>> " I have run it in one large cluster, so it is certified to be bug free/stable" I don't think it's a reasonable approach. We need as much feedback from our users because each of them stress Solr in a different way. This is not to suggest that committers are not doing testing or their tests are not valid. When I talk to the committers out here they say they do not see any performance stability issues at all. But, my client reports issues on a day to day basis.
>> >>> >>>>
>> >>> >>>>
>> >>> >>>>
>> >>> >>>> > Definitely publish a Docker image BTW -- it's the best way to try out any software.
>> >>> >>>>
>> >>> >>>> Docker is not a big requirement for large scale installations. Most of them already have their own install scripts. Availability of docker is not important for them. If a user is only encouraged to install Solr because of a docker image , most likely they are not running a large enough cluster
>> >>> >>>>
>> >>> >>>>
>> >>> >>>>
>> >>> >>>> On Tue, Oct 6, 2020, 6:30 AM David Smiley <[hidden email]> wrote:
>> >>> >>>>>
>> >>> >>>>> Thanks so much for your responses Ishan... I'm getting much more information in this thread than my attempts to get questions answered on the JIRA issue months ago.  And especially,  thank you for volunteering for the difficult porting efforts!
>> >>> >>>>>
>> >>> >>>>> Tomas said:
>> >>> >>>>>>
>> >>> >>>>>>  I do agree with the previous comments that calling it "Solr 10" (even with the "-alpha") would confuse users, maybe use "reference"? or maybe something in reference to SOLR-14788?
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> I have the opposite opinion.  This word "reference" is baffling to me despite whatever Mark's explanation is.  I like the justification Ishan gave for 10-alpha and I don't think I could re-phrase his justification any better.  *If* the release was _not_ official (thus wouldn't show up in the usual places anyone would look for a release), I think it would alleviate that confusion concern even more, although I think "alpha" ought to be enough of a signal not to use it without digging deeper on what's going on.
>> >>> >>>>>
>> >>> >>>>> Alex then Ishan said:
>> >>> >>>>>>
>> >>> >>>>>> > Maybe we could release it to
>> >>> >>>>>> > committers community first and dogfood it "internally"?
>> >>> >>>>>>
>> >>> >>>>>> Alex: It is meaningless. Committers don't run large scale installations. We barely even have time to take care of running unit tests before destabilizing our builds. We are not the right audience. However, we all can anyway check out the branch and start playing with it, even without a release. There are orgs that don't want to install any code that wasn't officially released; this release is geared towards them (to help us test this at their scale).
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> I don't think it can be said what committers do and don't do with regards to running Solr.  All of us would answer this differently and at different points in time.  From time to time, though not at present, I've been well positioned to try out a new version of Solr in a stage/test environment to see how it goes.  (Putting on my Salesforce metaphorical hat...) Even though I'm not able to deploy it in a realistic way today, I'm able to run a battery of tests to see if one of the features we depend on have changed or is broken.  That's useful feedback to an alpha release!  And even though I'm saying I'm not well positioned to try out some new Solr release in a production-ish setting now, it's something I could make a good case for internally since upgrades take a lot of effort where I work.  It's in our interest for SolrCloud to be very stable (of course).
>> >>> >>>>>
>> >>> >>>>> Regardless, I think what you're driving at Ishan is that you want an "official" release -- one that goes through the whole ceremony.  You believe that people would be more likely to use it.  I think all we need to do is announce (similar to a real release) that there is some unofficial alpha distribution and that we want to solicit your feedback -- basically, help us find bugs.  Definitely publish a Docker image BTW -- it's the best way to try out any software.  I'm -0 on doing an official release for alpha software because it's unnecessary to achieve the goals and somewhat confusing.  I think the Solr 4 alpha/beta situation was different -- it was not some fork a committer was maintaining; it was the master branch of its time, and it was destined to be the very next release, not some possible future release.
>> >>> >>>>>
>> >>> >>>>> ~ David Smiley
>> >>> >>>>> Apache Lucene/Solr Search Developer
>> >>> >>>>> http://www.linkedin.com/in/davidwsmiley
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> -----------------------------------------------------
>> >>> Noble Paul
>> >>>
>> >>> ---------------------------------------------------------------------
>> >>> To unsubscribe, e-mail: [hidden email]
>> >>> For additional commands, e-mail: [hidden email]
>> >>>
>>
>>
>> --
>> -----------------------------------------------------
>> Noble Paul
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]



--
Anshum Gupta
Reply | Threaded
Open this post in threaded view
|

Re: Solr Alpha (EA) release of Reference Branch

Varun Thacker-4


On Tue, Oct 6, 2020 at 7:45 PM Anshum Gupta <[hidden email]> wrote:
Thanks for initiating this discussion, Ishan. 

For the sake of making sure that we are all on the same page, let me summarize my understanding and take on this thread.

The current situation
Mark has a reference branch, which the folks who have looked at the branch, feel that it’s a much better, improved, reliable, and sustainable version of the current master i.e. take the same baseline and make it better. We would like to get those changes to the project, but aren’t sure about how to do so. Releasing the branch when it’s ready to go, as an alpha release will allow users to test it.

1. Is releasing the branch officially going to help us achieve the goal of having a well tested branch ?
2. Assuming #1 is true, do we as a community want to release the branch officially and assume responsibility?

I think so! We should all try to help out to the best we can.

3. What is our path forward after the release I.e. do we merge the branch into master or swap out current master.

What do we plan to do (options).
I feel there is a consensus on everyone wanting the best for the project and wanting Marks’ changes released. 

#1 - There are differing opinions, and I personally think we can have our test harnesses test the new branch, but I think most companies running Solr at scale would have concerns with taking up an alpha release and deploying it in production. The various tests that a bunch of folks are working on is our best bet at testing out the branch, in which case I’m not sure if we want an official release.

#2 - I feel that having an official release and having artifacts show up in maven central will confuse people. The 4.0 alpha release was very different in the sense that it was the same branch, the code wasn’t replacing anything existing but introducing a completely new feature i.e. SolrCloud.

#3 - I’m still unclear on how these changes will be released in terms of the community consensus. I’ve tried to merge parts of Marks’ effort from another time into master, but it’s very difficult, almost impossible to isolate and extract commits on the basis of coverage/features/etc. This is a lot of really great effort and after having spoken with Mark multiple times, I really feel we should figure out a way to absorb this but I do have concerns around replacing the master branch completely.

While I do like the idea that Tomás proposed, I also feel that maintaining and managing cherry-picking across 9x, master, and ref branch will only make it difficult for people to work though the duration of 9x. 

I haven’t looked at the current ref branch recently, but the folks who have looked at it, if you think that this code can be merged into master even as big chunks, that’d be the most confidence building way forward. 




On Tue, Oct 6, 2020 at 11:37 AM Ilan Ginzburg <[hidden email]> wrote:
Copying below Mark's posts from ASF Slack #solr-next-big-thing channel.

The Solr Reference Branch.
Document 1, a quick intro.
You can think of the Solr Reference Branch as a remaster of Solr. It
is not an attempt to redesign Solr or make it more fancy. The goal of
the Solr Reference Branch is to be a better incarnation of the current
Apache Solr, which will provide a base for future development and
design.
There are a variety of problems with Solr today that make it difficult
to adopt and run. This is me being as honest and objective as I can
be, though no doubt, many will see it as an exaggeration or negative
focus. I just see it as the way it is and has been, it's just taken me
a real long time to actually get all the way under the rug to find the
really hardened nasty cockroaches burrowed in there.
1. Resource usage and management is wasteful, inefficient, buggy, and haphazard.
2. SolrCloud is not long term reliable. Exceptional cases will
frequently flummox the system, and exceptional cases are supposed to
be our wheelhouse and primary focus. Leaders will be lost and not
recover, the Overseer will go away, GC storms will hit, tight loops in
a bad case will crank up resources, and retries will be abundant and
overaggressive.
3. Our blocking and locking is generally not efficient, especially in key paths.
4. We get thread safety wrong (too often) in some important spots.
5. Distributed updates have to be added locally before they are
distributed, and then that distribution is generally inefficient,
prone to blocking and/or timeouts, and hobbled by HTTP1.1 and our need
for packing updates into a single request to achieve any kind of
performance, losing proper error handling and eating the many rough
edges of the ConcurrentUpdateSolrClient.
6. Our Zookeeper foundation code is often inefficient, buggy,
unreliable, and improperly used (we don’t always use async or multi
where we should, we force updates from zk instead of being notified,
we don’t handle session expiration as well as we should, our
algorithms are slow and buggy, we make a multitude more calls than we
should (especially on cluster startup), etc, etc)
7. We have circular dependencies between major classes that can start
threads in their constructors that start interacting with the other
classes before construction is complete.
8. Our XML handling is abysmally outdated and slow for multiple
reasons. Our heavy Xpath usage is incredibly wasteful and expensive.
9. Our thread management is not understandable, not properly tunable,
not efficient, sometimes buggy, not always consistent, and difficult
to understand fundamentally.
10. Our Jetty configuration is lacking in a variety of ways,
especially around shutdown and http2.
11. The dynamic schema feature can be very expensive and not fully thread safe.
12. The Overseer is extremely inefficient, can be extremely slow to
stop, had a buggy leader election algorithm, doesn’t handle session
expiration as well as it should, can keep trying to come back from the
dead, and the list goes on.
13. Our connection resuse is often very poor or non existent, when
it’s improved, it always reverts back to bad or worse.
14. HTTP1.1 is not great for our type of application in a variety of
ways that HTTP2 solves – but we still use a lot of HTTP1.1 and HTTP2
is not configured well and the client needs some work.
15. Lifecycle of important objects is often off, most things can and
will leak (SolrCores, SolrIndexSearchers, Directory’s, Solr clients),
some things will close objects more than once or that don’t belong to
them, or close things in a bad order.
16. There is often sleeps and/or pulling that is a magnitude slower
than proper event driven waits.
17. Our tests are actually pretty unstable and making them stable is
way, way more difficult than most people realize. I’m quite sure I’ve
spent much, much more time on this than anyone out there, and I can
tell you, the tests are not stable in a 1,000 shifting ways that have
and will continue to cause lots of damage.
18. We don’t have good async update/search support for scaling and
better resource usage.
19. We often duplicate resources or create new pools instead of sharing.
20. We don’t do tons of parallelizable stuff in parallel, when we do
it’s inconsistent.
21. Our Collections API can often not wait correctly for the proper
state for what it did to be ready before returning. Even if it gets it
right, a cloud client that made the request won’t necessarily have the
updated state local when the request returns. Things often still work,
but with a variety of interesting and slow results possible.
22. We don’t often holistically look at what we have built and how it
fits together and so often there are silly things, bad fits, one off
bad patterns, lazy attempts at something, etc.
24. Close and shutdown are inefficient and slow across a huge swatch
of our object tree. These issues tend to be growy and breed less
concern over time.
25. There are a variety of ways and places that we can generate an
absurd amount of unnecessary garbage.
26. SolrCore reload is not fully reliable and increasingly important and used.
27. The leader election code has a variety of ugly little bugs and is
based on a recursive implementation that will eventually exhaust stack
space – though it’s likely your cluster will be brought down by
something else before that is a problem (unless you hit the infinite
loop, no one can be leader, eat up the stack as fast as possible case
– which should be hard these days with the leader election throttle).
28. The recovery processes, like almost everything you can imagine,
has a variety of issues and rarer bad cases and affects.
By and large, everything is inefficient and buggy and full of accepted
compromise regardless.
Interestingly, this does not make us an atypical open source Java
distributed project. But, I’m kind of a software snob, and I would not
run this thing and so I cannot work on it. What is there to do ...
The Solr Reference Branch is intended to tackle every one of those
issues. As well as about 1000+ more of varying and lesser importance.
As all of that comes together, cool stuff starts to unlock, and you
begin to see some phenomena that is together much greater than the sum
of it’s many, many parts.
29. Our tests have been getting better and better are stamping out the
legit noise they create - every scream a breadcrumb towards badness -
but we have built a scream catching machine - though we will never be
able to catch them all for a huge variety of deep reasons.

The Solr Reference Branch
Document 2
While the extent of the previously mentioned issues was not clear to
me, that is a deep rabbit hole, I’ve always, as have many others,
known the current state of things with Solr at a higher, broader
level.
So what about this effort is different? Is this not just a bunch of my
standard JIRA issues all crammed into one? Should we not break them
out proper and do things sensible?
Well, previously, as is probably common, I was both a bit lost on
where we were exactly and certainly on where to find firmer ground for
real, not just the mirage always just over the hill.
I love performance and efficiency though. I’ve always avoided it as a
focus with Solr and SolrCloud, thinking stability has to come first.
Having given up on stability and scale after a good 8 years or
something, completely tossed out as a pipe dream, I started work on
something new, something really just for me. I started plugging in
HTTP2. And the effort and work needed for that and the learning and
some of the results, completely opened my eyes. I also attacked very
different than I have in the past, doing something I like for me, I
drowned myself in it. Spent 2-3 weeks at a time here and there sitting
at the computer with intense focus for 16-20 hours a day. The more I
did, the more I found, the more I understood, the more I discovered.
I discovered a discovery processes. It was leading me to everything I
needed to do and I just had to follow the long, ever flowing path,
keeping my mental models strong, re etching, ruminating, obsessing.
I realized many test functions we have – most- should be taking on the
order of milliseconds instead of seconds to dozens of seconds. I
realized tons and tons of our issues and gremlins lived and prospered
in our slow and inefficient smog. I realized that if I just spent the
time to look where slowness and flakiness prevailed, really look, like
take hours just for some random side road - build a bridge, burn it,
and build one further down, etc, etc – that making huge improvement
after huge improvement was actually very low hanging fruit, just
hidden by some thorns and leaves and lack of any reasonable
introspection into the system we have created and continue to build.
Over time, I could see what had to be done and I could see what it
would achieve. I built different parts at different times, lost them
and rebuilt them a different way with different focus. I build and
expanded my introspection and measuring tools and classes.
That’s a sentence trying to cover a universe, but if you want to
really boil it down even further, I’d invoke the normally faulty
broken windows theory. There is magic in perfect windows that only
those that have them know. Can we get perfect? I like to dream and
there is no end to the introspection, experimentation, and
improvements to try. The perfect landing aside though, no doubt we can
move drastically from where we are.

Another thing I learned is the crazy number of ways you can make all
the tests pass like champions, and roll into production unusable.
Which tells me that production users are a large part of our test
strategy, and that can’t be to make any real change in a satisfactory
way.

The current goal is to have a mostly usable and testable system by
mid-late October. Not everything 100%, some known caveats and cleanup
and plenty to do, but it should be in good shape for a user to try out
given the caveats outlined
The biggest risk currently is the absorption of the search side async
work from master - I'm familiar with that, I've worked on it myself,
the code involved is derived from an old branch of mine, but async is
a whole different animal and trying to nail it without any downsides
to the old synchronous model is a tough nut
one that I was already battling on the dist update side, so it's good
stuff to work on and do, but its taking some effort to get in shape

On Tue, Oct 6, 2020 at 8:00 PM Tomás Fernández Löbbe
<[hidden email]> wrote:
>
> > Let's say we cut 9x and now there is a new master taken from the reference branch.
> I never said “make a new master”, I said merge changes in ref branch into master. If things are broken into pieces like Ishan is suggesting, those changes can be merged into 9.x too. I only suggested this because you felt unsure about merging to master now and I guess this is due to fear of introducing bugs so close to a potential 9.0 release, is that not right?
>
>
> > We will never be able to reconcile these 2 branches
> Sorry, but how is that different if we do an alpha release from the branch now? What would be the process after that? Let's say people don't find issues and we want to merge those changes, what’s the plan then?
>
> > Choice 1:
> I’m fine with choice 1 if that’s what you want, as long as it’s not an official release for the reasons stated above.
>
>
> > I promise to do code review & cleanup as much as possible. But I'm hesitant to give a stamp of approval to make it THE official release
> What do you mean? I thought this is what you were suggesting, make an official release from the reference_impl branch?
>
>
> I think Ilan’s last email is on spot, and I agree 100% with what he can express much better than I can :)
>
> > Mark's descriptions in Slack go in the right way but are still too high level
> Can someone share those here? or in Jira?
>
> On Tue, Oct 6, 2020 at 5:09 AM Noble Paul <[hidden email]> wrote:
>>
>> > I think the danger is high to treat this branch as a black box (or an "all or nothing").
>>
>> True Ilan.  Ideally, I would like a few of us to study the code &
>> start pulling in changes we are confident of (even to 8x branch, why
>> not). We cannot burden a single developer to do everything.
>>
>> This cannot be a task just for one or 2 devs. We all will have to work
>> together to decompose the changes and digest them into master. I can
>> do my bit.
>>
>> But, I'm sure we may hit a point where certain changes cannot be
>> isolated and absorbed. We will have to collectively make a call, how
>> to absorb them
>>
>> On Tue, Oct 6, 2020 at 9:00 PM Ishan Chattopadhyaya
>> <[hidden email]> wrote:
>> >
>> >
>> > I'm willing to help and I believe others will too if the amount of work for contributing is reasonable (i.e. not a three months effort).
>> >
>> > I looked into the possibility of doing so. To me, it seemed to be that it is very hard to do so: possibly 1 year project for me. Problem is that it is hard to pull out a particular class of improvements (say thread management improvement) and have all tests pass with it (because tests have gotten extensive improvements of their own) and also observe the effect of the improvement. IIUC, every improvement to Solr seemed to require many iterations to get the tests happy. I remember Mark telling me that it may not even be possible for him to do something like that (i.e. bring all changes into master as tiny pieces).
>> >
>> > What I volunteered to do, however, is to decompose roughly all the general improvements into smaller, manageable commits. However, making sure all tests pass at every commit point is beyond my capability.
>> >
>> > On Tue, 6 Oct, 2020, 3:10 pm Ilan Ginzburg, <[hidden email]> wrote:
>> >>
>> >> Another option to integrate this work into the main code line would be to understand what changes have been made and where (Mark's descriptions in Slack go in the right way but are still too high level), and then port or even redo them in main, one by one.
>> >>
>> >> I think the danger is high to treat this branch as a black box (or an "all or nothing"). Using the merging itself to change our understanding and increase our knowledge of what was done can greatly reduce the risk.
>> >>
>> >> We do develop new features in Solr 9 without beta releasing them, so if we port Mark's improvements by small chunks (and maybe in the process decide that some should not be ported or not now) I don't see why this can't integrate to become like other improvements done to the code. If specific changes do require a beta release, do that release from master and pick the right moment.
>> >>
>> >> I'm willing to help and I believe others will too if the amount of work for contributing is reasonable (i.e. not a three months effort). This requires documenting the changes done in that branch, pointing to where these changes happened and then picking them up one by one and porting them more or less independently of each other. We might only port a subset of changes by the time 9.0 is released, that's fine we can continue in following releases.
>> >>
>> >> My 2 cents...
>> >> Ilan
>> >>
>> >> Le mar. 6 oct. 2020 à 09:56, Noble Paul <[hidden email]> a écrit :
>> >>>
>> >>> Yes, A docker image will definitely help. I wasn't trying to downplay that
>> >>>
>> >>> On Tue, Oct 6, 2020 at 6:55 PM Ishan Chattopadhyaya
>> >>> <[hidden email]> wrote:
>> >>> >
>> >>> >
>> >>> > > Docker is not a big requirement for large scale installations. Most of them already have their own install scripts. Availability of docker is not important for them. If a user is only encouraged to install Solr because of a docker image , most likely they are not running a large enough cluster
>> >>> >
>> >>> > I disagree, Noble. Having a docker image us going to be useful to some clients, with complex usecases. Great point, David!
>> >>> >
>> >>> > On Tue, 6 Oct, 2020, 1:09 pm Ishan Chattopadhyaya, <[hidden email]> wrote:
>> >>> >>
>> >>> >> As I said, I'm *personally* not confident in putting such a big changeset into master that wasn't vetted in a real user environment widely. I have, in the past, done enough bad things to Solr (directly or indirectly), and I don't want to repeat the same. Also, I'll be very uncomfortable if someone else did so.
>> >>> >>
>> >>> >> Having said this, if someone else wants to port the changes over to master *without first getting enough real world testing*, feel free to do so, and I can focus my efforts elsewhere.
>> >>> >>
>> >>> >> On Tue, 6 Oct, 2020, 9:22 am Tomás Fernández Löbbe, <[hidden email]> wrote:
>> >>> >>>
>> >>> >>> I was thinking (and I haven’t flushed it out completely but will throw the idea) that an alternative approach with this timeline could be to cut 9x branch around November/December? And then you could merge into master, it would have the latest  changes from master plus the ref branch changes. From there any nightly build could be use to help test/debug.
>> >>> >>>
>> >>> >>> That said I don’t know for sure what are the changes in the branch that do not belong in 9. The problem with them being 10x only is that backports would potentially be more difficult for all the life of 9.
>> >>> >>>
>> >>> >>> On Mon, Oct 5, 2020 at 4:54 PM Noble Paul <[hidden email]> wrote:
>> >>> >>>>
>> >>> >>>> >I don't think it can be said what committers do and don't do with regards to running Solr.  All of us would answer this differently and at different points in time.
>> >>> >>>>
>> >>> >>>> " I have run it in one large cluster, so it is certified to be bug free/stable" I don't think it's a reasonable approach. We need as much feedback from our users because each of them stress Solr in a different way. This is not to suggest that committers are not doing testing or their tests are not valid. When I talk to the committers out here they say they do not see any performance stability issues at all. But, my client reports issues on a day to day basis.
>> >>> >>>>
>> >>> >>>>
>> >>> >>>>
>> >>> >>>> > Definitely publish a Docker image BTW -- it's the best way to try out any software.
>> >>> >>>>
>> >>> >>>> Docker is not a big requirement for large scale installations. Most of them already have their own install scripts. Availability of docker is not important for them. If a user is only encouraged to install Solr because of a docker image , most likely they are not running a large enough cluster
>> >>> >>>>
>> >>> >>>>
>> >>> >>>>
>> >>> >>>> On Tue, Oct 6, 2020, 6:30 AM David Smiley <[hidden email]> wrote:
>> >>> >>>>>
>> >>> >>>>> Thanks so much for your responses Ishan... I'm getting much more information in this thread than my attempts to get questions answered on the JIRA issue months ago.  And especially,  thank you for volunteering for the difficult porting efforts!
>> >>> >>>>>
>> >>> >>>>> Tomas said:
>> >>> >>>>>>
>> >>> >>>>>>  I do agree with the previous comments that calling it "Solr 10" (even with the "-alpha") would confuse users, maybe use "reference"? or maybe something in reference to SOLR-14788?
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> I have the opposite opinion.  This word "reference" is baffling to me despite whatever Mark's explanation is.  I like the justification Ishan gave for 10-alpha and I don't think I could re-phrase his justification any better.  *If* the release was _not_ official (thus wouldn't show up in the usual places anyone would look for a release), I think it would alleviate that confusion concern even more, although I think "alpha" ought to be enough of a signal not to use it without digging deeper on what's going on.
>> >>> >>>>>
>> >>> >>>>> Alex then Ishan said:
>> >>> >>>>>>
>> >>> >>>>>> > Maybe we could release it to
>> >>> >>>>>> > committers community first and dogfood it "internally"?
>> >>> >>>>>>
>> >>> >>>>>> Alex: It is meaningless. Committers don't run large scale installations. We barely even have time to take care of running unit tests before destabilizing our builds. We are not the right audience. However, we all can anyway check out the branch and start playing with it, even without a release. There are orgs that don't want to install any code that wasn't officially released; this release is geared towards them (to help us test this at their scale).
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> I don't think it can be said what committers do and don't do with regards to running Solr.  All of us would answer this differently and at different points in time.  From time to time, though not at present, I've been well positioned to try out a new version of Solr in a stage/test environment to see how it goes.  (Putting on my Salesforce metaphorical hat...) Even though I'm not able to deploy it in a realistic way today, I'm able to run a battery of tests to see if one of the features we depend on have changed or is broken.  That's useful feedback to an alpha release!  And even though I'm saying I'm not well positioned to try out some new Solr release in a production-ish setting now, it's something I could make a good case for internally since upgrades take a lot of effort where I work.  It's in our interest for SolrCloud to be very stable (of course).
>> >>> >>>>>
>> >>> >>>>> Regardless, I think what you're driving at Ishan is that you want an "official" release -- one that goes through the whole ceremony.  You believe that people would be more likely to use it.  I think all we need to do is announce (similar to a real release) that there is some unofficial alpha distribution and that we want to solicit your feedback -- basically, help us find bugs.  Definitely publish a Docker image BTW -- it's the best way to try out any software.  I'm -0 on doing an official release for alpha software because it's unnecessary to achieve the goals and somewhat confusing.  I think the Solr 4 alpha/beta situation was different -- it was not some fork a committer was maintaining; it was the master branch of its time, and it was destined to be the very next release, not some possible future release.
>> >>> >>>>>
>> >>> >>>>> ~ David Smiley
>> >>> >>>>> Apache Lucene/Solr Search Developer
>> >>> >>>>> http://www.linkedin.com/in/davidwsmiley
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> -----------------------------------------------------
>> >>> Noble Paul
>> >>>
>> >>> ---------------------------------------------------------------------
>> >>> To unsubscribe, e-mail: [hidden email]
>> >>> For additional commands, e-mail: [hidden email]
>> >>>
>>
>>
>> --
>> -----------------------------------------------------
>> Noble Paul
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]



--
Anshum Gupta
Reply | Threaded
Open this post in threaded view
|

Re: Solr Alpha (EA) release of Reference Branch

Robert Muir
In reply to this post by Anshum Gupta

On Tue, Oct 6, 2020 at 10:45 PM Anshum Gupta <[hidden email]> wrote:

I haven’t looked at the current ref branch recently, but the folks who have looked at it, if you think that this code can be merged into master even as big chunks, that’d be the most confidence building way forward. 


+1 for considering this approach. merge up master into the branch, and make a big-ass PR to merge back. let people help review (maybe improve) the change as a whole. It's just a big PR, some huge ones like this have been done in lucene before too, unofficially called "unfuck" branches (sorry if you are offended at my terminology). Sometimes you just fix, refactor, cleanup, and keep iterating and see where it can lead. sometimes you revert a bunch of commits because you followed the wrong rabbit-hole, etc.  Sometimes it may seem inconvenient, but I think we can all agree It's important to have folks that want to not just take the small fix, but see where it can go and make the whole thing better. Remember Mike's flexible indexing branch?

So why not try this way, look at actual code changes and try to get into the master branch? Of course Uwe is willing to point build resources at it either way, but if you want to maximize testing, start with the devs and everyone's jenkins first before throwing at users. Master branch will get you more testing, for sure.
Reply | Threaded
Open this post in threaded view
|

Re: Solr Alpha (EA) release of Reference Branch

Jan Høydahl / Cominvent
We want/need these improvements, for sure!
Agree treating this whole thing as a black box is dangerous. At the same time I realize that this will be one or a few huge merges anyway!
Rob’s suggestion is interesting! If at all possible to get the branch up to date with master, and make a PR, then it’s a great way for all of us to have a deeper look. 
And, who knows, perhaps end of this month there is consensus for merging into master sooner rather than later?
Especially if we have had reliably passing Jenkins runs of the new branch, with all tests enabled, for some time!
Risk is of course that 9.0.0 could be an unusually unstable release, but people don’t expect bug free x.0.0 releases. And to avoid cherry-pick hell for 6-12 months, perhaps that’s not a bad option after all.

Jan

7. okt. 2020 kl. 07:48 skrev Robert Muir <[hidden email]>:


On Tue, Oct 6, 2020 at 10:45 PM Anshum Gupta <[hidden email]> wrote:

I haven’t looked at the current ref branch recently, but the folks who have looked at it, if you think that this code can be merged into master even as big chunks, that’d be the most confidence building way forward. 


+1 for considering this approach. merge up master into the branch, and make a big-ass PR to merge back. let people help review (maybe improve) the change as a whole. It's just a big PR, some huge ones like this have been done in lucene before too, unofficially called "unfuck" branches (sorry if you are offended at my terminology). Sometimes you just fix, refactor, cleanup, and keep iterating and see where it can lead. sometimes you revert a bunch of commits because you followed the wrong rabbit-hole, etc.  Sometimes it may seem inconvenient, but I think we can all agree It's important to have folks that want to not just take the small fix, but see where it can go and make the whole thing better. Remember Mike's flexible indexing branch?

So why not try this way, look at actual code changes and try to get into the master branch? Of course Uwe is willing to point build resources at it either way, but if you want to maximize testing, start with the devs and everyone's jenkins first before throwing at users. Master branch will get you more testing, for sure.

Reply | Threaded
Open this post in threaded view
|

Re: Solr Alpha (EA) release of Reference Branch

Ilan Ginzburg
TBH, a PR with more than 1400 changed files is hard to look at. How many of us will invest a few weeks at least to really understand it? We should assume that if we don't bring these changes piece by piece, we risk having an unstable version of SolrCloud for a while.
When I look at some ongoing (unrelated) PR's judged too complex and being asked to split and simplify, this one would be at least two orders of magnitude "worse". 

Maybe this means releasing 9 without this branch but with all recent changes so we have a relatively stable release and merging the branch into master right afterwards as the base for Solr 10, then accepting a relatively long "baking" and bug fixing period?
A beta release of 10 might then make more sense, since it will be "release candidate" type code and not a separate branch that has diverged.

Ilan 

Le mer. 7 oct. 2020 à 13:43, Jan Høydahl <[hidden email]> a écrit :
We want/need these improvements, for sure!
Agree treating this whole thing as a black box is dangerous. At the same time I realize that this will be one or a few huge merges anyway!
Rob’s suggestion is interesting! If at all possible to get the branch up to date with master, and make a PR, then it’s a great way for all of us to have a deeper look. 
And, who knows, perhaps end of this month there is consensus for merging into master sooner rather than later?
Especially if we have had reliably passing Jenkins runs of the new branch, with all tests enabled, for some time!
Risk is of course that 9.0.0 could be an unusually unstable release, but people don’t expect bug free x.0.0 releases. And to avoid cherry-pick hell for 6-12 months, perhaps that’s not a bad option after all.

Jan

7. okt. 2020 kl. 07:48 skrev Robert Muir <[hidden email]>:


On Tue, Oct 6, 2020 at 10:45 PM Anshum Gupta <[hidden email]> wrote:

I haven’t looked at the current ref branch recently, but the folks who have looked at it, if you think that this code can be merged into master even as big chunks, that’d be the most confidence building way forward. 


+1 for considering this approach. merge up master into the branch, and make a big-ass PR to merge back. let people help review (maybe improve) the change as a whole. It's just a big PR, some huge ones like this have been done in lucene before too, unofficially called "unfuck" branches (sorry if you are offended at my terminology). Sometimes you just fix, refactor, cleanup, and keep iterating and see where it can lead. sometimes you revert a bunch of commits because you followed the wrong rabbit-hole, etc.  Sometimes it may seem inconvenient, but I think we can all agree It's important to have folks that want to not just take the small fix, but see where it can go and make the whole thing better. Remember Mike's flexible indexing branch?

So why not try this way, look at actual code changes and try to get into the master branch? Of course Uwe is willing to point build resources at it either way, but if you want to maximize testing, start with the devs and everyone's jenkins first before throwing at users. Master branch will get you more testing, for sure.

Reply | Threaded
Open this post in threaded view
|

Re: Solr Alpha (EA) release of Reference Branch

David Smiley
I think whatever changes can be safely brought from that branch into master should be done now independently of whatever happens to that branch.  It may not be a lot of this branch, but it's something.  For example off the top of my head -- pre-compiling XPath expressions.  AIUI, many changes cannot be ported because they purterb tests even if the change itself is fine.  So those can't be ported unless someone's willing to alo address the test failures that go with it.

~ David Smiley
Apache Lucene/Solr Search Developer


On Wed, Oct 7, 2020 at 9:36 AM Ilan Ginzburg <[hidden email]> wrote:
TBH, a PR with more than 1400 changed files is hard to look at. How many of us will invest a few weeks at least to really understand it? We should assume that if we don't bring these changes piece by piece, we risk having an unstable version of SolrCloud for a while.
When I look at some ongoing (unrelated) PR's judged too complex and being asked to split and simplify, this one would be at least two orders of magnitude "worse". 

Maybe this means releasing 9 without this branch but with all recent changes so we have a relatively stable release and merging the branch into master right afterwards as the base for Solr 10, then accepting a relatively long "baking" and bug fixing period?
A beta release of 10 might then make more sense, since it will be "release candidate" type code and not a separate branch that has diverged.

Ilan 

Le mer. 7 oct. 2020 à 13:43, Jan Høydahl <[hidden email]> a écrit :
We want/need these improvements, for sure!
Agree treating this whole thing as a black box is dangerous. At the same time I realize that this will be one or a few huge merges anyway!
Rob’s suggestion is interesting! If at all possible to get the branch up to date with master, and make a PR, then it’s a great way for all of us to have a deeper look. 
And, who knows, perhaps end of this month there is consensus for merging into master sooner rather than later?
Especially if we have had reliably passing Jenkins runs of the new branch, with all tests enabled, for some time!
Risk is of course that 9.0.0 could be an unusually unstable release, but people don’t expect bug free x.0.0 releases. And to avoid cherry-pick hell for 6-12 months, perhaps that’s not a bad option after all.

Jan

7. okt. 2020 kl. 07:48 skrev Robert Muir <[hidden email]>:


On Tue, Oct 6, 2020 at 10:45 PM Anshum Gupta <[hidden email]> wrote:

I haven’t looked at the current ref branch recently, but the folks who have looked at it, if you think that this code can be merged into master even as big chunks, that’d be the most confidence building way forward. 


+1 for considering this approach. merge up master into the branch, and make a big-ass PR to merge back. let people help review (maybe improve) the change as a whole. It's just a big PR, some huge ones like this have been done in lucene before too, unofficially called "unfuck" branches (sorry if you are offended at my terminology). Sometimes you just fix, refactor, cleanup, and keep iterating and see where it can lead. sometimes you revert a bunch of commits because you followed the wrong rabbit-hole, etc.  Sometimes it may seem inconvenient, but I think we can all agree It's important to have folks that want to not just take the small fix, but see where it can go and make the whole thing better. Remember Mike's flexible indexing branch?

So why not try this way, look at actual code changes and try to get into the master branch? Of course Uwe is willing to point build resources at it either way, but if you want to maximize testing, start with the devs and everyone's jenkins first before throwing at users. Master branch will get you more testing, for sure.

12