Separate git repo(s) for Solr modules

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Separate git repo(s) for Solr modules

Ishan Chattopadhyaya
Hi Devs,

As we discussed over the last few months, there seems a need to move non-core pieces away from the Solr core module. The contribs are presently a good place, but it makes sense to have a separate git repository hosting such modules. Some candidates that come to mind are the present day contrib modules, upcoming HDFS support module (separated away from solr-core), other first party packages. Along with that, there is also a need for a repository for hosting WIP modules/sub-projects.

I propose that we apply for the creation of two new git repositories:
1. solr-extras (or lucene-solr-extras)
2. solr-sandbox (or lucene-solr-sandbox)

Well tested, well supported modules/sub-projects can be released straight away from solr-extras. The first party packages can be built from this location and shipped with Solr (or be available for install using the package manager CLI).

New, unproved, beta, unstable modules can be hosted on solr-sandbox (and graduate to solr-extras once stable).

Please let me know if there are any questions/concerns with this approach.

Thanks and regards,
Ishan
Reply | Threaded
Open this post in threaded view
|

Re: Separate git repo(s) for Solr modules

Houston Putman
I think it is a good idea to have solr-extras and solr-sandbox. However, I think it's fine if some projects in solr-extras need to be migrated into their own repos at some point.

I don't necessarily agree with moving all contrib modules out of the main repo however. I think it makes the most sense to leave the "first-party" plugin contribs inside the repo, so that they are released at the same cadence as Solr itself, and compatibility is guaranteed whenever changes are made to Solr core. Anything not in the main repo or released in the same cadence as Solr is guaranteed to become stale with time. And if we do have "first-party" plugins, then it stands to reason that those are supposed to be supported on day-1 of a Solr release.

Some contrib modules aren't plugins, such as the Prometheus Exporter, so this would make sense to be moved to something like solr-extras.

- Houston

On Wed, Jan 13, 2021 at 10:30 AM Ishan Chattopadhyaya <[hidden email]> wrote:
Hi Devs,

As we discussed over the last few months, there seems a need to move non-core pieces away from the Solr core module. The contribs are presently a good place, but it makes sense to have a separate git repository hosting such modules. Some candidates that come to mind are the present day contrib modules, upcoming HDFS support module (separated away from solr-core), other first party packages. Along with that, there is also a need for a repository for hosting WIP modules/sub-projects.

I propose that we apply for the creation of two new git repositories:
1. solr-extras (or lucene-solr-extras)
2. solr-sandbox (or lucene-solr-sandbox)

Well tested, well supported modules/sub-projects can be released straight away from solr-extras. The first party packages can be built from this location and shipped with Solr (or be available for install using the package manager CLI).

New, unproved, beta, unstable modules can be hosted on solr-sandbox (and graduate to solr-extras once stable).

Please let me know if there are any questions/concerns with this approach.

Thanks and regards,
Ishan
Reply | Threaded
Open this post in threaded view
|

Re: Separate git repo(s) for Solr modules

Atri Sharma-3
In reply to this post by Ishan Chattopadhyaya
+1

This is also an opportunity to create the distinction between first party supported packages and the other plug-ins.



On Wed, 13 Jan 2021, 21:00 Ishan Chattopadhyaya, <[hidden email]> wrote:
Hi Devs,

As we discussed over the last few months, there seems a need to move non-core pieces away from the Solr core module. The contribs are presently a good place, but it makes sense to have a separate git repository hosting such modules. Some candidates that come to mind are the present day contrib modules, upcoming HDFS support module (separated away from solr-core), other first party packages. Along with that, there is also a need for a repository for hosting WIP modules/sub-projects.

I propose that we apply for the creation of two new git repositories:
1. solr-extras (or lucene-solr-extras)
2. solr-sandbox (or lucene-solr-sandbox)

Well tested, well supported modules/sub-projects can be released straight away from solr-extras. The first party packages can be built from this location and shipped with Solr (or be available for install using the package manager CLI).

New, unproved, beta, unstable modules can be hosted on solr-sandbox (and graduate to solr-extras once stable).

Please let me know if there are any questions/concerns with this approach.

Thanks and regards,
Ishan
Reply | Threaded
Open this post in threaded view
|

Re: Separate git repo(s) for Solr modules

Tomás Fernández Löbbe
Agere with Houston 100%. I think it's a good idea to have an official repo for things that are related to Solr and, at the same time, don't belong within the Solr codebase such as the prometheus exporter or, eventually, the cross-dc work that Anshum is working on (once it "graduates" from sandbox I guess). For the "first party" plugins (things like analysis-extra, langid, ltr), it's better to keep them together with the codebase, so that we can easily guarantee compatibility and releases together with Solr.

On Wed, Jan 13, 2021 at 8:26 AM Atri Sharma <[hidden email]> wrote:
+1

This is also an opportunity to create the distinction between first party supported packages and the other plug-ins.



On Wed, 13 Jan 2021, 21:00 Ishan Chattopadhyaya, <[hidden email]> wrote:
Hi Devs,

As we discussed over the last few months, there seems a need to move non-core pieces away from the Solr core module. The contribs are presently a good place, but it makes sense to have a separate git repository hosting such modules. Some candidates that come to mind are the present day contrib modules, upcoming HDFS support module (separated away from solr-core), other first party packages. Along with that, there is also a need for a repository for hosting WIP modules/sub-projects.

I propose that we apply for the creation of two new git repositories:
1. solr-extras (or lucene-solr-extras)
2. solr-sandbox (or lucene-solr-sandbox)

Well tested, well supported modules/sub-projects can be released straight away from solr-extras. The first party packages can be built from this location and shipped with Solr (or be available for install using the package manager CLI).

New, unproved, beta, unstable modules can be hosted on solr-sandbox (and graduate to solr-extras once stable).

Please let me know if there are any questions/concerns with this approach.

Thanks and regards,
Ishan
Reply | Threaded
Open this post in threaded view
|

Re: Separate git repo(s) for Solr modules

David Smiley
In reply to this post by Ishan Chattopadhyaya
Both paths have an appeal -- do contribs stay or move?  I recommend not moving existing contribs there until we explore how we can maintain compatibility with new Solr releases. (what Houston & Tomas point to below).  For example, perhaps a "git sub-module" pointing to Solr might be a key mechanism?  Could CI automatically keep it up to date?  Someone needs to try before we really know.

~ David Smiley
Apache Lucene/Solr Search Developer


On Wed, Jan 13, 2021 at 10:30 AM Ishan Chattopadhyaya <[hidden email]> wrote:
Hi Devs,

As we discussed over the last few months, there seems a need to move non-core pieces away from the Solr core module. The contribs are presently a good place, but it makes sense to have a separate git repository hosting such modules. Some candidates that come to mind are the present day contrib modules, upcoming HDFS support module (separated away from solr-core), other first party packages. Along with that, there is also a need for a repository for hosting WIP modules/sub-projects.

I propose that we apply for the creation of two new git repositories:
1. solr-extras (or lucene-solr-extras)
2. solr-sandbox (or lucene-solr-sandbox)

Well tested, well supported modules/sub-projects can be released straight away from solr-extras. The first party packages can be built from this location and shipped with Solr (or be available for install using the package manager CLI).

New, unproved, beta, unstable modules can be hosted on solr-sandbox (and graduate to solr-extras once stable).

Please let me know if there are any questions/concerns with this approach.

Thanks and regards,
Ishan
Reply | Threaded
Open this post in threaded view
|

Re: Separate git repo(s) for Solr modules

Ishan Chattopadhyaya
Sounds good, thanks Houston for bringing it up. I'll take a stab at a few first party packages soon and then we can take a call where they should belong (maybe on a case to case basis). 

Anshum has applied for the sandbox repo, I'll request for the extras repo soon.

Thanks!

On Thu, 14 Jan, 2021, 4:04 am David Smiley, <[hidden email]> wrote:
Both paths have an appeal -- do contribs stay or move?  I recommend not moving existing contribs there until we explore how we can maintain compatibility with new Solr releases. (what Houston & Tomas point to below).  For example, perhaps a "git sub-module" pointing to Solr might be a key mechanism?  Could CI automatically keep it up to date?  Someone needs to try before we really know.

~ David Smiley
Apache Lucene/Solr Search Developer


On Wed, Jan 13, 2021 at 10:30 AM Ishan Chattopadhyaya <[hidden email]> wrote:
Hi Devs,

As we discussed over the last few months, there seems a need to move non-core pieces away from the Solr core module. The contribs are presently a good place, but it makes sense to have a separate git repository hosting such modules. Some candidates that come to mind are the present day contrib modules, upcoming HDFS support module (separated away from solr-core), other first party packages. Along with that, there is also a need for a repository for hosting WIP modules/sub-projects.

I propose that we apply for the creation of two new git repositories:
1. solr-extras (or lucene-solr-extras)
2. solr-sandbox (or lucene-solr-sandbox)

Well tested, well supported modules/sub-projects can be released straight away from solr-extras. The first party packages can be built from this location and shipped with Solr (or be available for install using the package manager CLI).

New, unproved, beta, unstable modules can be hosted on solr-sandbox (and graduate to solr-extras once stable).

Please let me know if there are any questions/concerns with this approach.

Thanks and regards,
Ishan
Reply | Threaded
Open this post in threaded view
|

Re: Separate git repo(s) for Solr modules

Chris Hostetter-3
In reply to this post by Ishan Chattopadhyaya

: As we discussed over the last few months, there seems a need to move
: non-core pieces away from the Solr core module. The contribs are presently
: a good place, but it makes sense to have a separate git repository hosting
: such modules. Some candidates that come to mind are the present day contrib

can you explain why it makes sense to have a separate git repo for these
things?

I can think of lots of reasons why it makes sense to have a single
repo for all things solr (unified CI that quickly identifies if core
changes break "first order" plugins, shared feature branches & monotomic
commits of code that affects APIs and impls of those APIs, etc...) but
I've yet to see any concrete specifics of why multiple git repos are
"better" then just having distinct sub-projects (with distinct artifacts)
in the same repo other then "it makes sense"

why does it make sense?

why can't the ideas of "solr-sandbox" and "solr-extras" just be
directories in the "solr repo" ? ... what value is gained by making them
new repos?


-Hoss
http://www.lucidworks.com/

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]