Separate git repo(s) for Solr modules

classic Classic list List threaded Threaded
26 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Separate git repo(s) for Solr modules

Ishan Chattopadhyaya
Hi Devs,

As we discussed over the last few months, there seems a need to move non-core pieces away from the Solr core module. The contribs are presently a good place, but it makes sense to have a separate git repository hosting such modules. Some candidates that come to mind are the present day contrib modules, upcoming HDFS support module (separated away from solr-core), other first party packages. Along with that, there is also a need for a repository for hosting WIP modules/sub-projects.

I propose that we apply for the creation of two new git repositories:
1. solr-extras (or lucene-solr-extras)
2. solr-sandbox (or lucene-solr-sandbox)

Well tested, well supported modules/sub-projects can be released straight away from solr-extras. The first party packages can be built from this location and shipped with Solr (or be available for install using the package manager CLI).

New, unproved, beta, unstable modules can be hosted on solr-sandbox (and graduate to solr-extras once stable).

Please let me know if there are any questions/concerns with this approach.

Thanks and regards,
Ishan
Reply | Threaded
Open this post in threaded view
|

Re: Separate git repo(s) for Solr modules

Houston Putman
I think it is a good idea to have solr-extras and solr-sandbox. However, I think it's fine if some projects in solr-extras need to be migrated into their own repos at some point.

I don't necessarily agree with moving all contrib modules out of the main repo however. I think it makes the most sense to leave the "first-party" plugin contribs inside the repo, so that they are released at the same cadence as Solr itself, and compatibility is guaranteed whenever changes are made to Solr core. Anything not in the main repo or released in the same cadence as Solr is guaranteed to become stale with time. And if we do have "first-party" plugins, then it stands to reason that those are supposed to be supported on day-1 of a Solr release.

Some contrib modules aren't plugins, such as the Prometheus Exporter, so this would make sense to be moved to something like solr-extras.

- Houston

On Wed, Jan 13, 2021 at 10:30 AM Ishan Chattopadhyaya <[hidden email]> wrote:
Hi Devs,

As we discussed over the last few months, there seems a need to move non-core pieces away from the Solr core module. The contribs are presently a good place, but it makes sense to have a separate git repository hosting such modules. Some candidates that come to mind are the present day contrib modules, upcoming HDFS support module (separated away from solr-core), other first party packages. Along with that, there is also a need for a repository for hosting WIP modules/sub-projects.

I propose that we apply for the creation of two new git repositories:
1. solr-extras (or lucene-solr-extras)
2. solr-sandbox (or lucene-solr-sandbox)

Well tested, well supported modules/sub-projects can be released straight away from solr-extras. The first party packages can be built from this location and shipped with Solr (or be available for install using the package manager CLI).

New, unproved, beta, unstable modules can be hosted on solr-sandbox (and graduate to solr-extras once stable).

Please let me know if there are any questions/concerns with this approach.

Thanks and regards,
Ishan
Reply | Threaded
Open this post in threaded view
|

Re: Separate git repo(s) for Solr modules

Atri Sharma-3
In reply to this post by Ishan Chattopadhyaya
+1

This is also an opportunity to create the distinction between first party supported packages and the other plug-ins.



On Wed, 13 Jan 2021, 21:00 Ishan Chattopadhyaya, <[hidden email]> wrote:
Hi Devs,

As we discussed over the last few months, there seems a need to move non-core pieces away from the Solr core module. The contribs are presently a good place, but it makes sense to have a separate git repository hosting such modules. Some candidates that come to mind are the present day contrib modules, upcoming HDFS support module (separated away from solr-core), other first party packages. Along with that, there is also a need for a repository for hosting WIP modules/sub-projects.

I propose that we apply for the creation of two new git repositories:
1. solr-extras (or lucene-solr-extras)
2. solr-sandbox (or lucene-solr-sandbox)

Well tested, well supported modules/sub-projects can be released straight away from solr-extras. The first party packages can be built from this location and shipped with Solr (or be available for install using the package manager CLI).

New, unproved, beta, unstable modules can be hosted on solr-sandbox (and graduate to solr-extras once stable).

Please let me know if there are any questions/concerns with this approach.

Thanks and regards,
Ishan
Reply | Threaded
Open this post in threaded view
|

Re: Separate git repo(s) for Solr modules

Tomás Fernández Löbbe
Agere with Houston 100%. I think it's a good idea to have an official repo for things that are related to Solr and, at the same time, don't belong within the Solr codebase such as the prometheus exporter or, eventually, the cross-dc work that Anshum is working on (once it "graduates" from sandbox I guess). For the "first party" plugins (things like analysis-extra, langid, ltr), it's better to keep them together with the codebase, so that we can easily guarantee compatibility and releases together with Solr.

On Wed, Jan 13, 2021 at 8:26 AM Atri Sharma <[hidden email]> wrote:
+1

This is also an opportunity to create the distinction between first party supported packages and the other plug-ins.



On Wed, 13 Jan 2021, 21:00 Ishan Chattopadhyaya, <[hidden email]> wrote:
Hi Devs,

As we discussed over the last few months, there seems a need to move non-core pieces away from the Solr core module. The contribs are presently a good place, but it makes sense to have a separate git repository hosting such modules. Some candidates that come to mind are the present day contrib modules, upcoming HDFS support module (separated away from solr-core), other first party packages. Along with that, there is also a need for a repository for hosting WIP modules/sub-projects.

I propose that we apply for the creation of two new git repositories:
1. solr-extras (or lucene-solr-extras)
2. solr-sandbox (or lucene-solr-sandbox)

Well tested, well supported modules/sub-projects can be released straight away from solr-extras. The first party packages can be built from this location and shipped with Solr (or be available for install using the package manager CLI).

New, unproved, beta, unstable modules can be hosted on solr-sandbox (and graduate to solr-extras once stable).

Please let me know if there are any questions/concerns with this approach.

Thanks and regards,
Ishan
Reply | Threaded
Open this post in threaded view
|

Re: Separate git repo(s) for Solr modules

David Smiley
In reply to this post by Ishan Chattopadhyaya
Both paths have an appeal -- do contribs stay or move?  I recommend not moving existing contribs there until we explore how we can maintain compatibility with new Solr releases. (what Houston & Tomas point to below).  For example, perhaps a "git sub-module" pointing to Solr might be a key mechanism?  Could CI automatically keep it up to date?  Someone needs to try before we really know.

~ David Smiley
Apache Lucene/Solr Search Developer


On Wed, Jan 13, 2021 at 10:30 AM Ishan Chattopadhyaya <[hidden email]> wrote:
Hi Devs,

As we discussed over the last few months, there seems a need to move non-core pieces away from the Solr core module. The contribs are presently a good place, but it makes sense to have a separate git repository hosting such modules. Some candidates that come to mind are the present day contrib modules, upcoming HDFS support module (separated away from solr-core), other first party packages. Along with that, there is also a need for a repository for hosting WIP modules/sub-projects.

I propose that we apply for the creation of two new git repositories:
1. solr-extras (or lucene-solr-extras)
2. solr-sandbox (or lucene-solr-sandbox)

Well tested, well supported modules/sub-projects can be released straight away from solr-extras. The first party packages can be built from this location and shipped with Solr (or be available for install using the package manager CLI).

New, unproved, beta, unstable modules can be hosted on solr-sandbox (and graduate to solr-extras once stable).

Please let me know if there are any questions/concerns with this approach.

Thanks and regards,
Ishan
Reply | Threaded
Open this post in threaded view
|

Re: Separate git repo(s) for Solr modules

Ishan Chattopadhyaya
Sounds good, thanks Houston for bringing it up. I'll take a stab at a few first party packages soon and then we can take a call where they should belong (maybe on a case to case basis). 

Anshum has applied for the sandbox repo, I'll request for the extras repo soon.

Thanks!

On Thu, 14 Jan, 2021, 4:04 am David Smiley, <[hidden email]> wrote:
Both paths have an appeal -- do contribs stay or move?  I recommend not moving existing contribs there until we explore how we can maintain compatibility with new Solr releases. (what Houston & Tomas point to below).  For example, perhaps a "git sub-module" pointing to Solr might be a key mechanism?  Could CI automatically keep it up to date?  Someone needs to try before we really know.

~ David Smiley
Apache Lucene/Solr Search Developer


On Wed, Jan 13, 2021 at 10:30 AM Ishan Chattopadhyaya <[hidden email]> wrote:
Hi Devs,

As we discussed over the last few months, there seems a need to move non-core pieces away from the Solr core module. The contribs are presently a good place, but it makes sense to have a separate git repository hosting such modules. Some candidates that come to mind are the present day contrib modules, upcoming HDFS support module (separated away from solr-core), other first party packages. Along with that, there is also a need for a repository for hosting WIP modules/sub-projects.

I propose that we apply for the creation of two new git repositories:
1. solr-extras (or lucene-solr-extras)
2. solr-sandbox (or lucene-solr-sandbox)

Well tested, well supported modules/sub-projects can be released straight away from solr-extras. The first party packages can be built from this location and shipped with Solr (or be available for install using the package manager CLI).

New, unproved, beta, unstable modules can be hosted on solr-sandbox (and graduate to solr-extras once stable).

Please let me know if there are any questions/concerns with this approach.

Thanks and regards,
Ishan
Reply | Threaded
Open this post in threaded view
|

Re: Separate git repo(s) for Solr modules

Chris Hostetter-3
In reply to this post by Ishan Chattopadhyaya

: As we discussed over the last few months, there seems a need to move
: non-core pieces away from the Solr core module. The contribs are presently
: a good place, but it makes sense to have a separate git repository hosting
: such modules. Some candidates that come to mind are the present day contrib

can you explain why it makes sense to have a separate git repo for these
things?

I can think of lots of reasons why it makes sense to have a single
repo for all things solr (unified CI that quickly identifies if core
changes break "first order" plugins, shared feature branches & monotomic
commits of code that affects APIs and impls of those APIs, etc...) but
I've yet to see any concrete specifics of why multiple git repos are
"better" then just having distinct sub-projects (with distinct artifacts)
in the same repo other then "it makes sense"

why does it make sense?

why can't the ideas of "solr-sandbox" and "solr-extras" just be
directories in the "solr repo" ? ... what value is gained by making them
new repos?


-Hoss
http://www.lucidworks.com/

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Separate git repo(s) for Solr modules

Jason Gerlowski
Tentative +1 to Hoss' questions.  I agree with his summary of the
potential risks here, and I share his ignorance of the perceived
benefits.

(I thought for a time that this was driven by interest in having
release cadences independent from Solr-core releases.  I'm all for
that goal, but if that's the motivation I'm not sure what the obstacle
is to doing that with a single repo - all build systems these days
support versioning and releasing modules independent of one another.
But maybe that was never a driving factor here.)

I know there have been a lot of discussions about this, and I know the
repo has already been created.  So maybe it's too late to object even
if I wanted to, which I'm not sure I do.  But can someone that
understands the motivation please summarize what multiple-repos gets
us over a single repo that outweighs the "cons" that Hoss mentioned?

Jason

On Thu, Jan 14, 2021 at 12:34 PM Chris Hostetter
<[hidden email]> wrote:

>
>
> : As we discussed over the last few months, there seems a need to move
> : non-core pieces away from the Solr core module. The contribs are presently
> : a good place, but it makes sense to have a separate git repository hosting
> : such modules. Some candidates that come to mind are the present day contrib
>
> can you explain why it makes sense to have a separate git repo for these
> things?
>
> I can think of lots of reasons why it makes sense to have a single
> repo for all things solr (unified CI that quickly identifies if core
> changes break "first order" plugins, shared feature branches & monotomic
> commits of code that affects APIs and impls of those APIs, etc...) but
> I've yet to see any concrete specifics of why multiple git repos are
> "better" then just having distinct sub-projects (with distinct artifacts)
> in the same repo other then "it makes sense"
>
> why does it make sense?
>
> why can't the ideas of "solr-sandbox" and "solr-extras" just be
> directories in the "solr repo" ? ... what value is gained by making them
> new repos?
>
>
> -Hoss
> http://www.lucidworks.com/
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Separate git repo(s) for Solr modules

Ishan Chattopadhyaya
I haven't been able to follow up on creation of the extras repo, but more importantly I wanted to respond to Hoss. I'm out on an emergency for a week or so, shall resume then. If there's a decision on this until then, I shall accept it.

On Mon, 25 Jan, 2021, 9:04 am Jason Gerlowski, <[hidden email]> wrote:
Tentative +1 to Hoss' questions.  I agree with his summary of the
potential risks here, and I share his ignorance of the perceived
benefits.

(I thought for a time that this was driven by interest in having
release cadences independent from Solr-core releases.  I'm all for
that goal, but if that's the motivation I'm not sure what the obstacle
is to doing that with a single repo - all build systems these days
support versioning and releasing modules independent of one another.
But maybe that was never a driving factor here.)

I know there have been a lot of discussions about this, and I know the
repo has already been created.  So maybe it's too late to object even
if I wanted to, which I'm not sure I do.  But can someone that
understands the motivation please summarize what multiple-repos gets
us over a single repo that outweighs the "cons" that Hoss mentioned?

Jason

On Thu, Jan 14, 2021 at 12:34 PM Chris Hostetter
<[hidden email]> wrote:
>
>
> : As we discussed over the last few months, there seems a need to move
> : non-core pieces away from the Solr core module. The contribs are presently
> : a good place, but it makes sense to have a separate git repository hosting
> : such modules. Some candidates that come to mind are the present day contrib
>
> can you explain why it makes sense to have a separate git repo for these
> things?
>
> I can think of lots of reasons why it makes sense to have a single
> repo for all things solr (unified CI that quickly identifies if core
> changes break "first order" plugins, shared feature branches & monotomic
> commits of code that affects APIs and impls of those APIs, etc...) but
> I've yet to see any concrete specifics of why multiple git repos are
> "better" then just having distinct sub-projects (with distinct artifacts)
> in the same repo other then "it makes sense"
>
> why does it make sense?
>
> why can't the ideas of "solr-sandbox" and "solr-extras" just be
> directories in the "solr repo" ? ... what value is gained by making them
> new repos?
>
>
> -Hoss
> http://www.lucidworks.com/
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Separate git repo(s) for Solr modules

Ishan Chattopadhyaya
Hoss,
One important reason why separate repo for solr-extras is a good idea, as opposed to conrib modules, is that separate repo can be used to test against many Solr versions. Imagine a component that works across Solr versions 6x through 9x from day one. I can't imagine such a component being part of the lucene-solr repo itself.

Also, today, contrib modules are shipped with Solr, which we don't want for new things we might want to build.

On Mon, Jan 25, 2021 at 4:25 PM Ishan Chattopadhyaya <[hidden email]> wrote:
I haven't been able to follow up on creation of the extras repo, but more importantly I wanted to respond to Hoss. I'm out on an emergency for a week or so, shall resume then. If there's a decision on this until then, I shall accept it.

On Mon, 25 Jan, 2021, 9:04 am Jason Gerlowski, <[hidden email]> wrote:
Tentative +1 to Hoss' questions.  I agree with his summary of the
potential risks here, and I share his ignorance of the perceived
benefits.

(I thought for a time that this was driven by interest in having
release cadences independent from Solr-core releases.  I'm all for
that goal, but if that's the motivation I'm not sure what the obstacle
is to doing that with a single repo - all build systems these days
support versioning and releasing modules independent of one another.
But maybe that was never a driving factor here.)

I know there have been a lot of discussions about this, and I know the
repo has already been created.  So maybe it's too late to object even
if I wanted to, which I'm not sure I do.  But can someone that
understands the motivation please summarize what multiple-repos gets
us over a single repo that outweighs the "cons" that Hoss mentioned?

Jason

On Thu, Jan 14, 2021 at 12:34 PM Chris Hostetter
<[hidden email]> wrote:
>
>
> : As we discussed over the last few months, there seems a need to move
> : non-core pieces away from the Solr core module. The contribs are presently
> : a good place, but it makes sense to have a separate git repository hosting
> : such modules. Some candidates that come to mind are the present day contrib
>
> can you explain why it makes sense to have a separate git repo for these
> things?
>
> I can think of lots of reasons why it makes sense to have a single
> repo for all things solr (unified CI that quickly identifies if core
> changes break "first order" plugins, shared feature branches & monotomic
> commits of code that affects APIs and impls of those APIs, etc...) but
> I've yet to see any concrete specifics of why multiple git repos are
> "better" then just having distinct sub-projects (with distinct artifacts)
> in the same repo other then "it makes sense"
>
> why does it make sense?
>
> why can't the ideas of "solr-sandbox" and "solr-extras" just be
> directories in the "solr repo" ? ... what value is gained by making them
> new repos?
>
>
> -Hoss
> http://www.lucidworks.com/
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Separate git repo(s) for Solr modules

Jan Høydahl / Cominvent
Hi,

One important reason why separate repo for solr-extras is a good idea, as opposed to conrib modules, is that separate repo can be used to test against many Solr versions. Imagine a component that works across Solr versions 6x through 9x from day one. I can't imagine such a component being part of the lucene-solr repo itself.

For independent tools such as solr-operator or prometheus-exporter, or some future standalone DIH, it makes sense with separate repo and release cadence. However it will be much easier for our 1st party packages to remain in the monorepo, keeping all tests in-sync with core.

Also, today, contrib modules are shipped with Solr, which we don't want for new things we might want to build.

Remember that Solr main tarball does not need to include all our packages. We can easily extend our releae process to release e.g. solr-cell-9.5, solr-analysis-9.5, solr-auth-xyz-9.5 in our release repo together with the 9.5 release, i.e. release every package with every Solr version. Then we do not need a compatibility matrix, since users will be able to install the latest package. And each release can ship with a pre-installed package repo for that version.

I imagine it will be very very hard to provide cross-major-version compatibility for some plugins. It may get easier as more higher-level APIs are build to shield plugins from low-level inner code, but we're not there.

Jan


15. feb. 2021 kl. 15:17 skrev Ishan Chattopadhyaya <[hidden email]>:

Hoss,
One important reason why separate repo for solr-extras is a good idea, as opposed to conrib modules, is that separate repo can be used to test against many Solr versions. Imagine a component that works across Solr versions 6x through 9x from day one. I can't imagine such a component being part of the lucene-solr repo itself.

Also, today, contrib modules are shipped with Solr, which we don't want for new things we might want to build.

On Mon, Jan 25, 2021 at 4:25 PM Ishan Chattopadhyaya <[hidden email]> wrote:
I haven't been able to follow up on creation of the extras repo, but more importantly I wanted to respond to Hoss. I'm out on an emergency for a week or so, shall resume then. If there's a decision on this until then, I shall accept it.

On Mon, 25 Jan, 2021, 9:04 am Jason Gerlowski, <[hidden email]> wrote:
Tentative +1 to Hoss' questions.  I agree with his summary of the
potential risks here, and I share his ignorance of the perceived
benefits.

(I thought for a time that this was driven by interest in having
release cadences independent from Solr-core releases.  I'm all for
that goal, but if that's the motivation I'm not sure what the obstacle
is to doing that with a single repo - all build systems these days
support versioning and releasing modules independent of one another.
But maybe that was never a driving factor here.)

I know there have been a lot of discussions about this, and I know the
repo has already been created.  So maybe it's too late to object even
if I wanted to, which I'm not sure I do.  But can someone that
understands the motivation please summarize what multiple-repos gets
us over a single repo that outweighs the "cons" that Hoss mentioned?

Jason

On Thu, Jan 14, 2021 at 12:34 PM Chris Hostetter
<[hidden email]> wrote:
>
>
> : As we discussed over the last few months, there seems a need to move
> : non-core pieces away from the Solr core module. The contribs are presently
> : a good place, but it makes sense to have a separate git repository hosting
> : such modules. Some candidates that come to mind are the present day contrib
>
> can you explain why it makes sense to have a separate git repo for these
> things?
>
> I can think of lots of reasons why it makes sense to have a single
> repo for all things solr (unified CI that quickly identifies if core
> changes break "first order" plugins, shared feature branches & monotomic
> commits of code that affects APIs and impls of those APIs, etc...) but
> I've yet to see any concrete specifics of why multiple git repos are
> "better" then just having distinct sub-projects (with distinct artifacts)
> in the same repo other then "it makes sense"
>
> why does it make sense?
>
> why can't the ideas of "solr-sandbox" and "solr-extras" just be
> directories in the "solr repo" ? ... what value is gained by making them
> new repos?
>
>
> -Hoss
> http://www.lucidworks.com/
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: Separate git repo(s) for Solr modules

David Smiley
In reply to this post by Ishan Chattopadhyaya
I think embracing Docker is how we can identify if a plugin works in multiple versions of Solr.  The setup of such a test would start Solr via Docker and install the plugin into a Solr server using the package manager.  The test code itself would interact with Solr purely via SolrClient.  This would be an integration / smoke level test for the plugin.  I test my plugins at work similar to this way at a "smoke" level, which follows "integration".  What's cool is that we can take the very same test and either have it run via an embedded Solr (which we call an integration test) or a remote'ed Solr (run via Docker, which we call our "smoke" test).  Naturally, if a test makes assumptions that only work in embedded, then it won't run in smoke mode, so we have annotations to categorize the tests.  I'd like to work on some JUnit "Rule" for Solr, similar to what I have at work (and have done in the past) -- https://issues.apache.org/jira/browse/SOLR-11872 

~ David Smiley
Apache Lucene/Solr Search Developer


On Mon, Feb 15, 2021 at 9:17 AM Ishan Chattopadhyaya <[hidden email]> wrote:
Hoss,
One important reason why separate repo for solr-extras is a good idea, as opposed to conrib modules, is that separate repo can be used to test against many Solr versions. Imagine a component that works across Solr versions 6x through 9x from day one. I can't imagine such a component being part of the lucene-solr repo itself.

Also, today, contrib modules are shipped with Solr, which we don't want for new things we might want to build.

On Mon, Jan 25, 2021 at 4:25 PM Ishan Chattopadhyaya <[hidden email]> wrote:
I haven't been able to follow up on creation of the extras repo, but more importantly I wanted to respond to Hoss. I'm out on an emergency for a week or so, shall resume then. If there's a decision on this until then, I shall accept it.

On Mon, 25 Jan, 2021, 9:04 am Jason Gerlowski, <[hidden email]> wrote:
Tentative +1 to Hoss' questions.  I agree with his summary of the
potential risks here, and I share his ignorance of the perceived
benefits.

(I thought for a time that this was driven by interest in having
release cadences independent from Solr-core releases.  I'm all for
that goal, but if that's the motivation I'm not sure what the obstacle
is to doing that with a single repo - all build systems these days
support versioning and releasing modules independent of one another.
But maybe that was never a driving factor here.)

I know there have been a lot of discussions about this, and I know the
repo has already been created.  So maybe it's too late to object even
if I wanted to, which I'm not sure I do.  But can someone that
understands the motivation please summarize what multiple-repos gets
us over a single repo that outweighs the "cons" that Hoss mentioned?

Jason

On Thu, Jan 14, 2021 at 12:34 PM Chris Hostetter
<[hidden email]> wrote:
>
>
> : As we discussed over the last few months, there seems a need to move
> : non-core pieces away from the Solr core module. The contribs are presently
> : a good place, but it makes sense to have a separate git repository hosting
> : such modules. Some candidates that come to mind are the present day contrib
>
> can you explain why it makes sense to have a separate git repo for these
> things?
>
> I can think of lots of reasons why it makes sense to have a single
> repo for all things solr (unified CI that quickly identifies if core
> changes break "first order" plugins, shared feature branches & monotomic
> commits of code that affects APIs and impls of those APIs, etc...) but
> I've yet to see any concrete specifics of why multiple git repos are
> "better" then just having distinct sub-projects (with distinct artifacts)
> in the same repo other then "it makes sense"
>
> why does it make sense?
>
> why can't the ideas of "solr-sandbox" and "solr-extras" just be
> directories in the "solr repo" ? ... what value is gained by making them
> new repos?
>
>
> -Hoss
> http://www.lucidworks.com/
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Separate git repo(s) for Solr modules

Jason Gerlowski
> separate repo can be used to test against many Solr versions. Imagine a component that works across Solr versions 6x through 9x from day one. I can't imagine such a component being part of the lucene-solr repo itself.

It'll be great to test contrib's on multiple Solr versions, but I
can't see any blockers for this that are same-repo specific.  Can you
elaborate on what makes this tricky in a same-repo setup?  AFAICT you
can set up multiple gradle test tasks that rely on different
solr-test-framework versions and run tests that way - regardless of
where the code lives.   (Or alternatively, docker images for various
Solr versions as David suggested.)

I think Jan's got a good point about how do-able it'll be in practice
to have plugins support multiple major release versions.  That said
though, a lot of these questions (docker tests vs otherwise, cross
major-version plugins?) will likely sort themselves out when it comes
time to implement.

On Tue, Feb 16, 2021 at 12:21 AM David Smiley <[hidden email]> wrote:

>
> I think embracing Docker is how we can identify if a plugin works in multiple versions of Solr.  The setup of such a test would start Solr via Docker and install the plugin into a Solr server using the package manager.  The test code itself would interact with Solr purely via SolrClient.  This would be an integration / smoke level test for the plugin.  I test my plugins at work similar to this way at a "smoke" level, which follows "integration".  What's cool is that we can take the very same test and either have it run via an embedded Solr (which we call an integration test) or a remote'ed Solr (run via Docker, which we call our "smoke" test).  Naturally, if a test makes assumptions that only work in embedded, then it won't run in smoke mode, so we have annotations to categorize the tests.  I'd like to work on some JUnit "Rule" for Solr, similar to what I have at work (and have done in the past) -- https://issues.apache.org/jira/browse/SOLR-11872
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
>
> On Mon, Feb 15, 2021 at 9:17 AM Ishan Chattopadhyaya <[hidden email]> wrote:
>>
>> Hoss,
>> One important reason why separate repo for solr-extras is a good idea, as opposed to conrib modules, is that separate repo can be used to test against many Solr versions. Imagine a component that works across Solr versions 6x through 9x from day one. I can't imagine such a component being part of the lucene-solr repo itself.
>>
>> Also, today, contrib modules are shipped with Solr, which we don't want for new things we might want to build.
>>
>> On Mon, Jan 25, 2021 at 4:25 PM Ishan Chattopadhyaya <[hidden email]> wrote:
>>>
>>> I haven't been able to follow up on creation of the extras repo, but more importantly I wanted to respond to Hoss. I'm out on an emergency for a week or so, shall resume then. If there's a decision on this until then, I shall accept it.
>>>
>>> On Mon, 25 Jan, 2021, 9:04 am Jason Gerlowski, <[hidden email]> wrote:
>>>>
>>>> Tentative +1 to Hoss' questions.  I agree with his summary of the
>>>> potential risks here, and I share his ignorance of the perceived
>>>> benefits.
>>>>
>>>> (I thought for a time that this was driven by interest in having
>>>> release cadences independent from Solr-core releases.  I'm all for
>>>> that goal, but if that's the motivation I'm not sure what the obstacle
>>>> is to doing that with a single repo - all build systems these days
>>>> support versioning and releasing modules independent of one another.
>>>> But maybe that was never a driving factor here.)
>>>>
>>>> I know there have been a lot of discussions about this, and I know the
>>>> repo has already been created.  So maybe it's too late to object even
>>>> if I wanted to, which I'm not sure I do.  But can someone that
>>>> understands the motivation please summarize what multiple-repos gets
>>>> us over a single repo that outweighs the "cons" that Hoss mentioned?
>>>>
>>>> Jason
>>>>
>>>> On Thu, Jan 14, 2021 at 12:34 PM Chris Hostetter
>>>> <[hidden email]> wrote:
>>>> >
>>>> >
>>>> > : As we discussed over the last few months, there seems a need to move
>>>> > : non-core pieces away from the Solr core module. The contribs are presently
>>>> > : a good place, but it makes sense to have a separate git repository hosting
>>>> > : such modules. Some candidates that come to mind are the present day contrib
>>>> >
>>>> > can you explain why it makes sense to have a separate git repo for these
>>>> > things?
>>>> >
>>>> > I can think of lots of reasons why it makes sense to have a single
>>>> > repo for all things solr (unified CI that quickly identifies if core
>>>> > changes break "first order" plugins, shared feature branches & monotomic
>>>> > commits of code that affects APIs and impls of those APIs, etc...) but
>>>> > I've yet to see any concrete specifics of why multiple git repos are
>>>> > "better" then just having distinct sub-projects (with distinct artifacts)
>>>> > in the same repo other then "it makes sense"
>>>> >
>>>> > why does it make sense?
>>>> >
>>>> > why can't the ideas of "solr-sandbox" and "solr-extras" just be
>>>> > directories in the "solr repo" ? ... what value is gained by making them
>>>> > new repos?
>>>> >
>>>> >
>>>> > -Hoss
>>>> > http://www.lucidworks.com/
>>>> >
>>>> > ---------------------------------------------------------------------
>>>> > To unsubscribe, e-mail: [hidden email]
>>>> > For additional commands, e-mail: [hidden email]
>>>> >
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: [hidden email]
>>>> For additional commands, e-mail: [hidden email]
>>>>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Separate git repo(s) for Solr modules

Ishan Chattopadhyaya
Imagine a component that wants to have its own release cadence. Say, a 1.x line of the component that supports certain features (across Solr 6x to 9x) and 2.x line that supports certain other features (which are a big upgrade from previous 1.x). A Solr UI can be a good example, where say 1.x uses React and 2.x uses AngularJS.
Shoving such a component into lucene-solr repo makes no sense, given its branching strategy is based on master / branch_8x (and later it will become branch_9x and master).

On Tue, Feb 16, 2021 at 5:19 PM Jason Gerlowski <[hidden email]> wrote:
> separate repo can be used to test against many Solr versions. Imagine a component that works across Solr versions 6x through 9x from day one. I can't imagine such a component being part of the lucene-solr repo itself.

It'll be great to test contrib's on multiple Solr versions, but I
can't see any blockers for this that are same-repo specific.  Can you
elaborate on what makes this tricky in a same-repo setup?  AFAICT you
can set up multiple gradle test tasks that rely on different
solr-test-framework versions and run tests that way - regardless of
where the code lives.   (Or alternatively, docker images for various
Solr versions as David suggested.)

I think Jan's got a good point about how do-able it'll be in practice
to have plugins support multiple major release versions.  That said
though, a lot of these questions (docker tests vs otherwise, cross
major-version plugins?) will likely sort themselves out when it comes
time to implement.

On Tue, Feb 16, 2021 at 12:21 AM David Smiley <[hidden email]> wrote:
>
> I think embracing Docker is how we can identify if a plugin works in multiple versions of Solr.  The setup of such a test would start Solr via Docker and install the plugin into a Solr server using the package manager.  The test code itself would interact with Solr purely via SolrClient.  This would be an integration / smoke level test for the plugin.  I test my plugins at work similar to this way at a "smoke" level, which follows "integration".  What's cool is that we can take the very same test and either have it run via an embedded Solr (which we call an integration test) or a remote'ed Solr (run via Docker, which we call our "smoke" test).  Naturally, if a test makes assumptions that only work in embedded, then it won't run in smoke mode, so we have annotations to categorize the tests.  I'd like to work on some JUnit "Rule" for Solr, similar to what I have at work (and have done in the past) -- https://issues.apache.org/jira/browse/SOLR-11872
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
>
> On Mon, Feb 15, 2021 at 9:17 AM Ishan Chattopadhyaya <[hidden email]> wrote:
>>
>> Hoss,
>> One important reason why separate repo for solr-extras is a good idea, as opposed to conrib modules, is that separate repo can be used to test against many Solr versions. Imagine a component that works across Solr versions 6x through 9x from day one. I can't imagine such a component being part of the lucene-solr repo itself.
>>
>> Also, today, contrib modules are shipped with Solr, which we don't want for new things we might want to build.
>>
>> On Mon, Jan 25, 2021 at 4:25 PM Ishan Chattopadhyaya <[hidden email]> wrote:
>>>
>>> I haven't been able to follow up on creation of the extras repo, but more importantly I wanted to respond to Hoss. I'm out on an emergency for a week or so, shall resume then. If there's a decision on this until then, I shall accept it.
>>>
>>> On Mon, 25 Jan, 2021, 9:04 am Jason Gerlowski, <[hidden email]> wrote:
>>>>
>>>> Tentative +1 to Hoss' questions.  I agree with his summary of the
>>>> potential risks here, and I share his ignorance of the perceived
>>>> benefits.
>>>>
>>>> (I thought for a time that this was driven by interest in having
>>>> release cadences independent from Solr-core releases.  I'm all for
>>>> that goal, but if that's the motivation I'm not sure what the obstacle
>>>> is to doing that with a single repo - all build systems these days
>>>> support versioning and releasing modules independent of one another.
>>>> But maybe that was never a driving factor here.)
>>>>
>>>> I know there have been a lot of discussions about this, and I know the
>>>> repo has already been created.  So maybe it's too late to object even
>>>> if I wanted to, which I'm not sure I do.  But can someone that
>>>> understands the motivation please summarize what multiple-repos gets
>>>> us over a single repo that outweighs the "cons" that Hoss mentioned?
>>>>
>>>> Jason
>>>>
>>>> On Thu, Jan 14, 2021 at 12:34 PM Chris Hostetter
>>>> <[hidden email]> wrote:
>>>> >
>>>> >
>>>> > : As we discussed over the last few months, there seems a need to move
>>>> > : non-core pieces away from the Solr core module. The contribs are presently
>>>> > : a good place, but it makes sense to have a separate git repository hosting
>>>> > : such modules. Some candidates that come to mind are the present day contrib
>>>> >
>>>> > can you explain why it makes sense to have a separate git repo for these
>>>> > things?
>>>> >
>>>> > I can think of lots of reasons why it makes sense to have a single
>>>> > repo for all things solr (unified CI that quickly identifies if core
>>>> > changes break "first order" plugins, shared feature branches & monotomic
>>>> > commits of code that affects APIs and impls of those APIs, etc...) but
>>>> > I've yet to see any concrete specifics of why multiple git repos are
>>>> > "better" then just having distinct sub-projects (with distinct artifacts)
>>>> > in the same repo other then "it makes sense"
>>>> >
>>>> > why does it make sense?
>>>> >
>>>> > why can't the ideas of "solr-sandbox" and "solr-extras" just be
>>>> > directories in the "solr repo" ? ... what value is gained by making them
>>>> > new repos?
>>>> >
>>>> >
>>>> > -Hoss
>>>> > http://www.lucidworks.com/
>>>> >
>>>> > ---------------------------------------------------------------------
>>>> > To unsubscribe, e-mail: [hidden email]
>>>> > For additional commands, e-mail: [hidden email]
>>>> >
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: [hidden email]
>>>> For additional commands, e-mail: [hidden email]
>>>>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Separate git repo(s) for Solr modules

Eric Pugh-4
Testing across multiple versions is always very difficult ;-).  I recently saw this very interesting approach to using our Dockerized Solr’s to test a component against a number of previous versions of Solr.  https://github.com/querqy/querqy/pull/147. I’m hopeful it could be a model for other packages to adopt.

Trying to maintain across multiple versions is kind of a Sisyphean task, and I don’t think plays to open source communities strengths.   It’s hard enough to keep all components of Solr up to date with the latest Lucene and the latest Solr….  (Try out wt=xlsx Response Writer, it’s currently broken on master) .  Reminds me of the Apache Gump project ;-)

If something is really going to be backcompatible across certain versions, then maybe having it’s own repo makes sense, but I suspect it means those components may go stale….   Look at DIH and Velocity components that are moved over to their own repos, both are getting stale, and I’d argue it’s because they don’t live in the main project where all of us have oversight and easy access.


On Feb 16, 2021, at 7:44 AM, Ishan Chattopadhyaya <[hidden email]> wrote:

Imagine a component that wants to have its own release cadence. Say, a 1.x line of the component that supports certain features (across Solr 6x to 9x) and 2.x line that supports certain other features (which are a big upgrade from previous 1.x). A Solr UI can be a good example, where say 1.x uses React and 2.x uses AngularJS.
Shoving such a component into lucene-solr repo makes no sense, given its branching strategy is based on master / branch_8x (and later it will become branch_9x and master).

On Tue, Feb 16, 2021 at 5:19 PM Jason Gerlowski <[hidden email]> wrote:
> separate repo can be used to test against many Solr versions. Imagine a component that works across Solr versions 6x through 9x from day one. I can't imagine such a component being part of the lucene-solr repo itself.

It'll be great to test contrib's on multiple Solr versions, but I
can't see any blockers for this that are same-repo specific.  Can you
elaborate on what makes this tricky in a same-repo setup?  AFAICT you
can set up multiple gradle test tasks that rely on different
solr-test-framework versions and run tests that way - regardless of
where the code lives.   (Or alternatively, docker images for various
Solr versions as David suggested.)

I think Jan's got a good point about how do-able it'll be in practice
to have plugins support multiple major release versions.  That said
though, a lot of these questions (docker tests vs otherwise, cross
major-version plugins?) will likely sort themselves out when it comes
time to implement.

On Tue, Feb 16, 2021 at 12:21 AM David Smiley <[hidden email]> wrote:
>
> I think embracing Docker is how we can identify if a plugin works in multiple versions of Solr.  The setup of such a test would start Solr via Docker and install the plugin into a Solr server using the package manager.  The test code itself would interact with Solr purely via SolrClient.  This would be an integration / smoke level test for the plugin.  I test my plugins at work similar to this way at a "smoke" level, which follows "integration".  What's cool is that we can take the very same test and either have it run via an embedded Solr (which we call an integration test) or a remote'ed Solr (run via Docker, which we call our "smoke" test).  Naturally, if a test makes assumptions that only work in embedded, then it won't run in smoke mode, so we have annotations to categorize the tests.  I'd like to work on some JUnit "Rule" for Solr, similar to what I have at work (and have done in the past) -- https://issues.apache.org/jira/browse/SOLR-11872
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
>
> On Mon, Feb 15, 2021 at 9:17 AM Ishan Chattopadhyaya <[hidden email]> wrote:
>>
>> Hoss,
>> One important reason why separate repo for solr-extras is a good idea, as opposed to conrib modules, is that separate repo can be used to test against many Solr versions. Imagine a component that works across Solr versions 6x through 9x from day one. I can't imagine such a component being part of the lucene-solr repo itself.
>>
>> Also, today, contrib modules are shipped with Solr, which we don't want for new things we might want to build.
>>
>> On Mon, Jan 25, 2021 at 4:25 PM Ishan Chattopadhyaya <[hidden email]> wrote:
>>>
>>> I haven't been able to follow up on creation of the extras repo, but more importantly I wanted to respond to Hoss. I'm out on an emergency for a week or so, shall resume then. If there's a decision on this until then, I shall accept it.
>>>
>>> On Mon, 25 Jan, 2021, 9:04 am Jason Gerlowski, <[hidden email]> wrote:
>>>>
>>>> Tentative +1 to Hoss' questions.  I agree with his summary of the
>>>> potential risks here, and I share his ignorance of the perceived
>>>> benefits.
>>>>
>>>> (I thought for a time that this was driven by interest in having
>>>> release cadences independent from Solr-core releases.  I'm all for
>>>> that goal, but if that's the motivation I'm not sure what the obstacle
>>>> is to doing that with a single repo - all build systems these days
>>>> support versioning and releasing modules independent of one another.
>>>> But maybe that was never a driving factor here.)
>>>>
>>>> I know there have been a lot of discussions about this, and I know the
>>>> repo has already been created.  So maybe it's too late to object even
>>>> if I wanted to, which I'm not sure I do.  But can someone that
>>>> understands the motivation please summarize what multiple-repos gets
>>>> us over a single repo that outweighs the "cons" that Hoss mentioned?
>>>>
>>>> Jason
>>>>
>>>> On Thu, Jan 14, 2021 at 12:34 PM Chris Hostetter
>>>> <[hidden email]> wrote:
>>>> >
>>>> >
>>>> > : As we discussed over the last few months, there seems a need to move
>>>> > : non-core pieces away from the Solr core module. The contribs are presently
>>>> > : a good place, but it makes sense to have a separate git repository hosting
>>>> > : such modules. Some candidates that come to mind are the present day contrib
>>>> >
>>>> > can you explain why it makes sense to have a separate git repo for these
>>>> > things?
>>>> >
>>>> > I can think of lots of reasons why it makes sense to have a single
>>>> > repo for all things solr (unified CI that quickly identifies if core
>>>> > changes break "first order" plugins, shared feature branches & monotomic
>>>> > commits of code that affects APIs and impls of those APIs, etc...) but
>>>> > I've yet to see any concrete specifics of why multiple git repos are
>>>> > "better" then just having distinct sub-projects (with distinct artifacts)
>>>> > in the same repo other then "it makes sense"
>>>> >
>>>> > why does it make sense?
>>>> >
>>>> > why can't the ideas of "solr-sandbox" and "solr-extras" just be
>>>> > directories in the "solr repo" ? ... what value is gained by making them
>>>> > new repos?
>>>> >
>>>> >
>>>> > -Hoss
>>>> > http://www.lucidworks.com/
>>>> >
>>>> > ---------------------------------------------------------------------
>>>> > To unsubscribe, e-mail: [hidden email]
>>>> > For additional commands, e-mail: [hidden email]
>>>> >
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: [hidden email]
>>>> For additional commands, e-mail: [hidden email]
>>>>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


_______________________
Eric Pugh Founder & CEO | OpenSource Connections, LLC | 434.466.1467 | http://www.opensourceconnections.com | My Free/Busy  
This e-mail and all contents, including attachments, is considered to be Company Confidential unless explicitly stated otherwise, regardless of whether attachments are marked as such.

Reply | Threaded
Open this post in threaded view
|

Re: Separate git repo(s) for Solr modules

David Smiley
On Tue, Feb 16, 2021 at 8:38 AM Eric Pugh <[hidden email]> wrote:
Testing across multiple versions is always very difficult ;-).  I recently saw this very interesting approach to using our Dockerized Solr’s to test a component against a number of previous versions of Solr.  https://github.com/querqy/querqy/pull/147. I’m hopeful it could be a model for other packages to adopt.

Thanks for the link to that Querqy PR.  That is *very* similar to what I do at work (minus multi-version testing), and also similar to how I test multiple versions in one of my pet projects by using a CI build matrix of a configurable dependency.  I didn't know Testcontainer.org has its own Solr module -- https://www.testcontainers.org/modules/solr/ but we implemented that ourselves; not hard.
 
Trying to maintain across multiple versions is kind of a Sisyphean task, and I don’t think plays to open source communities strengths.   It’s hard enough to keep all components of Solr up to date with the latest Lucene and the latest Solr….  (Try out wt=xlsx Response Writer, it’s currently broken on master) .  Reminds me of the Apache Gump project ;-)

If something is really going to be backcompatible across certain versions, then maybe having it’s own repo makes sense,

I'd prefer that a module only leave our "contribs" area when the concerns/limitations become real.  Doing it prematurely could lead to atrophy of the module....
 
but I suspect it means those components may go stale….   Look at DIH and Velocity components that are moved over to their own repos, both are getting stale, and I’d argue it’s because they don’t live in the main project where all of us have oversight and easy access.

Agreed :-(
 
Reply | Threaded
Open this post in threaded view
|

Re: Separate git repo(s) for Solr modules

Jason Gerlowski
> Shoving such a component into lucene-solr repo makes no sense, given its branching strategy is based on master / branch_8x

I get how this could cause issues - I def hadn't thought much about
multi-version support and branching.  But I don't think moving plugins
to a separate repo solves that problem for us.  If our first class
plugins are set up to have different release "lines" that don't line
up with major Solr versions, it's only a matter of time before two of
those plugins have branch requirements that conflict.  Unless I'm
missing something here?

> I'd prefer that a module only leave our "contribs" area when the concerns/limitations become real.  Doing it prematurely could lead to atrophy of the module....

+1 to David's comment.   I def hadn't considered the branching-scheme
issues that Ishan brought up in his last reply, and they seem like
valid concerns to me.  But the risk and the downsides of "atrophy" are
serious enough that I'd vote to not risk them until this starts to
cause us issues in practice.  Even if, for now, that means we won't be
able to build a single plugin jar that supports (e.g.) 3 major Solr
versions.  IMO that's a much smaller loss.

On Tue, Feb 16, 2021 at 9:40 AM David Smiley <[hidden email]> wrote:

>
> On Tue, Feb 16, 2021 at 8:38 AM Eric Pugh <[hidden email]> wrote:
>>
>> Testing across multiple versions is always very difficult ;-).  I recently saw this very interesting approach to using our Dockerized Solr’s to test a component against a number of previous versions of Solr.  https://github.com/querqy/querqy/pull/147. I’m hopeful it could be a model for other packages to adopt.
>
>
> Thanks for the link to that Querqy PR.  That is *very* similar to what I do at work (minus multi-version testing), and also similar to how I test multiple versions in one of my pet projects by using a CI build matrix of a configurable dependency.  I didn't know Testcontainer.org has its own Solr module -- https://www.testcontainers.org/modules/solr/ but we implemented that ourselves; not hard.
>
>>
>> Trying to maintain across multiple versions is kind of a Sisyphean task, and I don’t think plays to open source communities strengths.   It’s hard enough to keep all components of Solr up to date with the latest Lucene and the latest Solr….  (Try out wt=xlsx Response Writer, it’s currently broken on master) .  Reminds me of the Apache Gump project ;-)
>>
>> If something is really going to be backcompatible across certain versions, then maybe having it’s own repo makes sense,
>
>
> I'd prefer that a module only leave our "contribs" area when the concerns/limitations become real.  Doing it prematurely could lead to atrophy of the module....
>
>>
>> but I suspect it means those components may go stale….   Look at DIH and Velocity components that are moved over to their own repos, both are getting stale, and I’d argue it’s because they don’t live in the main project where all of us have oversight and easy access.
>
>
> Agreed :-(
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Separate git repo(s) for Solr modules

Noble Paul നോബിള്‍  नोब्ळ्
I'm still struggling to understand the workflow when I'm working on a feature that spans lucene and solr. 

I'm yet to see an argument against sub-modules

On Wed, Feb 17, 2021 at 3:18 AM Jason Gerlowski <[hidden email]> wrote:
> Shoving such a component into lucene-solr repo makes no sense, given its branching strategy is based on master / branch_8x

I get how this could cause issues - I def hadn't thought much about
multi-version support and branching.  But I don't think moving plugins
to a separate repo solves that problem for us.  If our first class
plugins are set up to have different release "lines" that don't line
up with major Solr versions, it's only a matter of time before two of
those plugins have branch requirements that conflict.  Unless I'm
missing something here?

> I'd prefer that a module only leave our "contribs" area when the concerns/limitations become real.  Doing it prematurely could lead to atrophy of the module....

+1 to David's comment.   I def hadn't considered the branching-scheme
issues that Ishan brought up in his last reply, and they seem like
valid concerns to me.  But the risk and the downsides of "atrophy" are
serious enough that I'd vote to not risk them until this starts to
cause us issues in practice.  Even if, for now, that means we won't be
able to build a single plugin jar that supports (e.g.) 3 major Solr
versions.  IMO that's a much smaller loss.

On Tue, Feb 16, 2021 at 9:40 AM David Smiley <[hidden email]> wrote:
>
> On Tue, Feb 16, 2021 at 8:38 AM Eric Pugh <[hidden email]> wrote:
>>
>> Testing across multiple versions is always very difficult ;-).  I recently saw this very interesting approach to using our Dockerized Solr’s to test a component against a number of previous versions of Solr.  https://github.com/querqy/querqy/pull/147. I’m hopeful it could be a model for other packages to adopt.
>
>
> Thanks for the link to that Querqy PR.  That is *very* similar to what I do at work (minus multi-version testing), and also similar to how I test multiple versions in one of my pet projects by using a CI build matrix of a configurable dependency.  I didn't know Testcontainer.org has its own Solr module -- https://www.testcontainers.org/modules/solr/ but we implemented that ourselves; not hard.
>
>>
>> Trying to maintain across multiple versions is kind of a Sisyphean task, and I don’t think plays to open source communities strengths.   It’s hard enough to keep all components of Solr up to date with the latest Lucene and the latest Solr….  (Try out wt=xlsx Response Writer, it’s currently broken on master) .  Reminds me of the Apache Gump project ;-)
>>
>> If something is really going to be backcompatible across certain versions, then maybe having it’s own repo makes sense,
>
>
> I'd prefer that a module only leave our "contribs" area when the concerns/limitations become real.  Doing it prematurely could lead to atrophy of the module....
>
>>
>> but I suspect it means those components may go stale….   Look at DIH and Velocity components that are moved over to their own repos, both are getting stale, and I’d argue it’s because they don’t live in the main project where all of us have oversight and easy access.
>
>
> Agreed :-(
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]



--
-----------------------------------------------------
Noble Paul
Reply | Threaded
Open this post in threaded view
|

Re: Separate git repo(s) for Solr modules

Houston Putman
In the future we wont be able to “work on both at the same time”, once Lucene 9 is cut. Why not pull that bandaid now?

On Mon, May 3, 2021 at 11:32 PM Noble Paul <[hidden email]> wrote:
I'm still struggling to understand the workflow when I'm working on a feature that spans lucene and solr. 

I'm yet to see an argument against sub-modules

On Wed, Feb 17, 2021 at 3:18 AM Jason Gerlowski <[hidden email]> wrote:
> Shoving such a component into lucene-solr repo makes no sense, given its branching strategy is based on master / branch_8x

I get how this could cause issues - I def hadn't thought much about
multi-version support and branching.  But I don't think moving plugins
to a separate repo solves that problem for us.  If our first class
plugins are set up to have different release "lines" that don't line
up with major Solr versions, it's only a matter of time before two of
those plugins have branch requirements that conflict.  Unless I'm
missing something here?

> I'd prefer that a module only leave our "contribs" area when the concerns/limitations become real.  Doing it prematurely could lead to atrophy of the module....

+1 to David's comment.   I def hadn't considered the branching-scheme
issues that Ishan brought up in his last reply, and they seem like
valid concerns to me.  But the risk and the downsides of "atrophy" are
serious enough that I'd vote to not risk them until this starts to
cause us issues in practice.  Even if, for now, that means we won't be
able to build a single plugin jar that supports (e.g.) 3 major Solr
versions.  IMO that's a much smaller loss.

On Tue, Feb 16, 2021 at 9:40 AM David Smiley <[hidden email]> wrote:
>
> On Tue, Feb 16, 2021 at 8:38 AM Eric Pugh <[hidden email]> wrote:
>>
>> Testing across multiple versions is always very difficult ;-).  I recently saw this very interesting approach to using our Dockerized Solr’s to test a component against a number of previous versions of Solr.  https://github.com/querqy/querqy/pull/147. I’m hopeful it could be a model for other packages to adopt.
>
>
> Thanks for the link to that Querqy PR.  That is *very* similar to what I do at work (minus multi-version testing), and also similar to how I test multiple versions in one of my pet projects by using a CI build matrix of a configurable dependency.  I didn't know Testcontainer.org has its own Solr module -- https://www.testcontainers.org/modules/solr/ but we implemented that ourselves; not hard.
>
>>
>> Trying to maintain across multiple versions is kind of a Sisyphean task, and I don’t think plays to open source communities strengths.   It’s hard enough to keep all components of Solr up to date with the latest Lucene and the latest Solr….  (Try out wt=xlsx Response Writer, it’s currently broken on master) .  Reminds me of the Apache Gump project ;-)
>>
>> If something is really going to be backcompatible across certain versions, then maybe having it’s own repo makes sense,
>
>
> I'd prefer that a module only leave our "contribs" area when the concerns/limitations become real.  Doing it prematurely could lead to atrophy of the module....
>
>>
>> but I suspect it means those components may go stale….   Look at DIH and Velocity components that are moved over to their own repos, both are getting stale, and I’d argue it’s because they don’t live in the main project where all of us have oversight and easy access.
>
>
> Agreed :-(
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]



--
-----------------------------------------------------
Noble Paul
Reply | Threaded
Open this post in threaded view
|

Re: Separate git repo(s) for Solr modules

Noble Paul നോബിള്‍  नोब्ळ्
@Houston

So, Are you suggesting we should not do that ? 

On Tue, May 4, 2021 at 2:35 PM Houston Putman <[hidden email]> wrote:
In the future we wont be able to “work on both at the same time”, once Lucene 9 is cut. Why not pull that bandaid now?

On Mon, May 3, 2021 at 11:32 PM Noble Paul <[hidden email]> wrote:
I'm still struggling to understand the workflow when I'm working on a feature that spans lucene and solr. 

I'm yet to see an argument against sub-modules

On Wed, Feb 17, 2021 at 3:18 AM Jason Gerlowski <[hidden email]> wrote:
> Shoving such a component into lucene-solr repo makes no sense, given its branching strategy is based on master / branch_8x

I get how this could cause issues - I def hadn't thought much about
multi-version support and branching.  But I don't think moving plugins
to a separate repo solves that problem for us.  If our first class
plugins are set up to have different release "lines" that don't line
up with major Solr versions, it's only a matter of time before two of
those plugins have branch requirements that conflict.  Unless I'm
missing something here?

> I'd prefer that a module only leave our "contribs" area when the concerns/limitations become real.  Doing it prematurely could lead to atrophy of the module....

+1 to David's comment.   I def hadn't considered the branching-scheme
issues that Ishan brought up in his last reply, and they seem like
valid concerns to me.  But the risk and the downsides of "atrophy" are
serious enough that I'd vote to not risk them until this starts to
cause us issues in practice.  Even if, for now, that means we won't be
able to build a single plugin jar that supports (e.g.) 3 major Solr
versions.  IMO that's a much smaller loss.

On Tue, Feb 16, 2021 at 9:40 AM David Smiley <[hidden email]> wrote:
>
> On Tue, Feb 16, 2021 at 8:38 AM Eric Pugh <[hidden email]> wrote:
>>
>> Testing across multiple versions is always very difficult ;-).  I recently saw this very interesting approach to using our Dockerized Solr’s to test a component against a number of previous versions of Solr.  https://github.com/querqy/querqy/pull/147. I’m hopeful it could be a model for other packages to adopt.
>
>
> Thanks for the link to that Querqy PR.  That is *very* similar to what I do at work (minus multi-version testing), and also similar to how I test multiple versions in one of my pet projects by using a CI build matrix of a configurable dependency.  I didn't know Testcontainer.org has its own Solr module -- https://www.testcontainers.org/modules/solr/ but we implemented that ourselves; not hard.
>
>>
>> Trying to maintain across multiple versions is kind of a Sisyphean task, and I don’t think plays to open source communities strengths.   It’s hard enough to keep all components of Solr up to date with the latest Lucene and the latest Solr….  (Try out wt=xlsx Response Writer, it’s currently broken on master) .  Reminds me of the Apache Gump project ;-)
>>
>> If something is really going to be backcompatible across certain versions, then maybe having it’s own repo makes sense,
>
>
> I'd prefer that a module only leave our "contribs" area when the concerns/limitations become real.  Doing it prematurely could lead to atrophy of the module....
>
>>
>> but I suspect it means those components may go stale….   Look at DIH and Velocity components that are moved over to their own repos, both are getting stale, and I’d argue it’s because they don’t live in the main project where all of us have oversight and easy access.
>
>
> Agreed :-(
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]



--
-----------------------------------------------------
Noble Paul


--
-----------------------------------------------------
Noble Paul
12