[DISCUSS] SIP-4 Solr should own the bootstrap process

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

[DISCUSS] SIP-4 Solr should own the bootstrap process

Jan Høydahl / Cominvent
It has been proposed many times to make Solr a truly standalone app and that we should start Jetty from within Solr and not the other way around :)
Here is my formal SIP proposal for this:

https://cwiki.apache.org/confluence/display/SOLR/SIP-4+Solr+should+own+the+bootstrap+process

Please read the SIP description and then come back here for discussion. As the discussion progresses, we can update the SIP page, and when there seems to be consensus around the approach, we can call a vote and move the SIP to the next table "Adopted/Accepted but unreleased SIPs».

Jan
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] SIP-4 Solr should own the bootstrap process

Uwe Schindler
Thanks Jan,

I fully agree. You already started with the work, now getting rid of start.jar should be easy.

I have some other ideas to slim down the distribution: I would put all static HTML and JavaScript into a JAR file and hook it up as Jetty ClassPathResource. Of course that's more complicated during development. but we can allow editing files in filesystem in the dev environment, but when the bootstrapper in production startup sees the static asset jar file in classpath, it could use it instead.

Should I add this to the wiki page?

Uwe

Am March 24, 2020 3:01:43 PM UTC schrieb "Jan Høydahl" <[hidden email]>:
It has been proposed many times to make Solr a truly standalone app and that we should start Jetty from within Solr and not the other way around :)
Here is my formal SIP proposal for this:

https://cwiki.apache.org/confluence/display/SOLR/SIP-4+Solr+should+own+the+bootstrap+process

Please read the SIP description and then come back here for discussion. As the discussion progresses, we can update the SIP page, and when there seems to be consensus around the approach, we can call a vote and move the SIP to the next table "Adopted/Accepted but unreleased SIPs».

Jan
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


--
Uwe Schindler
Achterdiek 19, 28357 Bremen
https://www.thetaphi.de
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] SIP-4 Solr should own the bootstrap process

Jan Høydahl / Cominvent
Thanks Uwe,

I’m not sure what should be the scope of SIP-4. It can grow almost as much as we want it to since it will enable so many new things. But probably wise to contain it to the bootstrapper module, including memory locking, getting rid of web.xml and perhaps a few more low-hanging fruits? Then it will be easy to do further work either in new SIP’s if they require architectural discussion, or as plain JIRAs.

But feel free to add your idea to the bullet list of what this will enable. Then we can define the exact scope of the SIP as we continue.

For those who have not followed the issues@ list, https://issues.apache.org/jira/browse/SOLR-14335 is the issue I have started already to create the bootstrap module and memory locking.

Jan

24. mar. 2020 kl. 16:09 skrev Uwe Schindler <[hidden email]>:

Thanks Jan,

I fully agree. You already started with the work, now getting rid of start.jar should be easy.

I have some other ideas to slim down the distribution: I would put all static HTML and JavaScript into a JAR file and hook it up as Jetty ClassPathResource. Of course that's more complicated during development. but we can allow editing files in filesystem in the dev environment, but when the bootstrapper in production startup sees the static asset jar file in classpath, it could use it instead.

Should I add this to the wiki page?

Uwe

Am March 24, 2020 3:01:43 PM UTC schrieb "Jan Høydahl" <[hidden email]>:
It has been proposed many times to make Solr a truly standalone app and that we should start Jetty from within Solr and not the other way around :)
Here is my formal SIP proposal for this:

https://cwiki.apache.org/confluence/display/SOLR/SIP-4+Solr+should+own+the+bootstrap+process

Please read the SIP description and then come back here for discussion. As the discussion progresses, we can update the SIP page, and when there seems to be consensus around the approach, we can call a vote and move the SIP to the next table "Adopted/Accepted but unreleased SIPs».

Jan
To unsubscribe, [hidden email]
For additional commands, [hidden email]


--
Uwe Schindler
Achterdiek 19, 28357 Bremen
https://www.thetaphi.de

Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] SIP-4 Solr should own the bootstrap process

Gus Heck
This SIP would also make a unification of timeout configuration much easier (SOLR-13457)

On Tue, Mar 24, 2020 at 11:24 AM Jan Høydahl <[hidden email]> wrote:
Thanks Uwe,

I’m not sure what should be the scope of SIP-4. It can grow almost as much as we want it to since it will enable so many new things. But probably wise to contain it to the bootstrapper module, including memory locking, getting rid of web.xml and perhaps a few more low-hanging fruits? Then it will be easy to do further work either in new SIP’s if they require architectural discussion, or as plain JIRAs.

But feel free to add your idea to the bullet list of what this will enable. Then we can define the exact scope of the SIP as we continue.

For those who have not followed the issues@ list, https://issues.apache.org/jira/browse/SOLR-14335 is the issue I have started already to create the bootstrap module and memory locking.

Jan

24. mar. 2020 kl. 16:09 skrev Uwe Schindler <[hidden email]>:

Thanks Jan,

I fully agree. You already started with the work, now getting rid of start.jar should be easy.

I have some other ideas to slim down the distribution: I would put all static HTML and JavaScript into a JAR file and hook it up as Jetty ClassPathResource. Of course that's more complicated during development. but we can allow editing files in filesystem in the dev environment, but when the bootstrapper in production startup sees the static asset jar file in classpath, it could use it instead.

Should I add this to the wiki page?

Uwe

Am March 24, 2020 3:01:43 PM UTC schrieb "Jan Høydahl" <[hidden email]>:
It has been proposed many times to make Solr a truly standalone app and that we should start Jetty from within Solr and not the other way around :)
Here is my formal SIP proposal for this:

https://cwiki.apache.org/confluence/display/SOLR/SIP-4+Solr+should+own+the+bootstrap+process

Please read the SIP description and then come back here for discussion. As the discussion progresses, we can update the SIP page, and when there seems to be consensus around the approach, we can call a vote and move the SIP to the next table "Adopted/Accepted but unreleased SIPs».

Jan
To unsubscribe, [hidden email]
For additional commands, [hidden email]


--
Uwe Schindler
Achterdiek 19, 28357 Bremen
https://www.thetaphi.de



--
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] SIP-4 Solr should own the bootstrap process

Bram Van Dam
In reply to this post by Jan Høydahl / Cominvent
On 24/03/2020 16:01, Jan Høydahl wrote:
> It has been proposed many times to make Solr a truly standalone app and that we should start Jetty from within Solr and not the other way around :)

Do you see this running in a single JVM, i.e. you run solr.jar and it
then starts an embedded Jetty server? Or would solr.jar exec a new Java
process and take it from there?

If it's all in one JVM, then the startup scripts probably won't be all
that different. Memory settings, OOM killers etc will still have to be
set up before using startup shellscripts and whatnot. If it's just a
small bootstrap jar which execs another JVM, a lot of the complicated
shell foo could probably be simplified.

 - Bram

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] SIP-4 Solr should own the bootstrap process

Uwe Schindler
Of course it's a single jvm. It's only a replacement for start.jar. the mock-up is already there.

Uwe

Am March 24, 2020 9:45:33 PM UTC schrieb Bram Van Dam <[hidden email]>:
On 24/03/2020 16:01, Jan Høydahl wrote:
It has been proposed many times to make Solr a truly standalone app and that we should start Jetty from within Solr and not the other way around :)

Do you see this running in a single JVM, i.e. you run solr.jar and it
then starts an embedded Jetty server? Or would solr.jar exec a new Java
process and take it from there?

If it's all in one JVM, then the startup scripts probably won't be all
that different. Memory settings, OOM killers etc will still have to be
set up before using startup shellscripts and whatnot. If it's just a
small bootstrap jar which execs another JVM, a lot of the complicated
shell foo could probably be simplified.

- Bram
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


--
Uwe Schindler
Achterdiek 19, 28357 Bremen
https://www.thetaphi.de
Reply | Threaded
Open this post in threaded view
|

[DISCUSS] SIP-6 Solr should own the bootstrap process

Jan Høydahl / Cominvent
This SIP has changed name to SIP-6 and I have changed the email subject of this thread, please keep this new subject going forward.
New SIP link: https://cwiki.apache.org/confluence/display/SOLR/SIP-6+Solr+should+own+the+bootstrap+process

Jan

24. mar. 2020 kl. 22:48 skrev Uwe Schindler <[hidden email]>:

Of course it's a single jvm. It's only a replacement for start.jar. the mock-up is already there.

Uwe

Am March 24, 2020 9:45:33 PM UTC schrieb Bram Van Dam <[hidden email]>:
On 24/03/2020 16:01, Jan Høydahl wrote:
It has been proposed many times to make Solr a truly standalone app and that we should start Jetty from within Solr and not the other way around :)

Do you see this running in a single JVM, i.e. you run solr.jar and it
then starts an embedded Jetty server? Or would solr.jar exec a new Java
process and take it from there?

If it's all in one JVM, then the startup scripts probably won't be all
that different. Memory settings, OOM killers etc will still have to be
set up before using startup shellscripts and whatnot. If it's just a
small bootstrap jar which execs another JVM, a lot of the complicated
shell foo could probably be simplified.

- Bram
To unsubscribe, [hidden email]
For additional commands, [hidden email]


--
Uwe Schindler
Achterdiek 19, 28357 Bremen
https://www.thetaphi.de

Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] SIP-6 Solr should own the bootstrap process

Mike Drob-3
I'm in favor of this approach mainly because of the line mentioned in the Test Plan:

> If we move servlet initialization into Java code and get rid of web.xml, we may be able to start Solr in the same way both in tests and in real life!


I feel that this should be a top level goal, along with taking another look at EmbeddedSolrServer (perhaps it moves out of SolrJ?). Right now, I feel like we effectively have no testing of the configurations that go into the various xml files - can't speak for others, but it made me reluctant to make changes to them.


Mike

On Tue, Mar 24, 2020 at 6:36 PM Jan Høydahl <[hidden email]> wrote:
This SIP has changed name to SIP-6 and I have changed the email subject of this thread, please keep this new subject going forward.
New SIP link: https://cwiki.apache.org/confluence/display/SOLR/SIP-6+Solr+should+own+the+bootstrap+process

Jan

24. mar. 2020 kl. 22:48 skrev Uwe Schindler <[hidden email]>:

Of course it's a single jvm. It's only a replacement for start.jar. the mock-up is already there.

Uwe

Am March 24, 2020 9:45:33 PM UTC schrieb Bram Van Dam <[hidden email]>:
On 24/03/2020 16:01, Jan Høydahl wrote:
It has been proposed many times to make Solr a truly standalone app and that we should start Jetty from within Solr and not the other way around :)

Do you see this running in a single JVM, i.e. you run solr.jar and it
then starts an embedded Jetty server? Or would solr.jar exec a new Java
process and take it from there?

If it's all in one JVM, then the startup scripts probably won't be all
that different. Memory settings, OOM killers etc will still have to be
set up before using startup shellscripts and whatnot. If it's just a
small bootstrap jar which execs another JVM, a lot of the complicated
shell foo could probably be simplified.

- Bram
To unsubscribe, [hidden email]
For additional commands, [hidden email]


--
Uwe Schindler
Achterdiek 19, 28357 Bremen
https://www.thetaphi.de

Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] SIP-6 Solr should own the bootstrap process

Jan Høydahl / Cominvent
I agree. I changed the SIP page and now have four bullet points defined as in scope for the SIP:

The most important change in this SIP will be

  • Create a solr/bootstrap module with a solr.jar  to replace start.jar  as the entrypoint (java -jar solr.jar ...)
  • Hide jetty even more, i.e. rename jetty.port  → port , and perhaps clean up other Sysprops as well
  • Get rid of web.xml and instead build our contexts and paths from Java code, driven by Solr config
  • Be able to lock Solr's memory to prevent swapping (SOLR-14335)

Not sure how big of a job the web.xml thing is, but there is definitely a potential of unifying test and production code in this area and thus get much better test coverage of some of this.

However, removing web.xml is perhaps such a big change that we’ll have to target 9.0 while my initial goal of just creating a bootstrapper to lock memory could have been merged to 8.x as well?? Or do we consider web.xml an implementation detail for which we don’t need to provide backward compatibility?

I’m a bit back and forth with myself on this. Perhaps if we reduce scope to just the bootstrap module, mem locking and some sysprop renames, it can be done quickly and end up in 8.6 already, then increment from there?

Jan

25. mar. 2020 kl. 16:13 skrev Mike Drob <[hidden email]>:

I'm in favor of this approach mainly because of the line mentioned in the Test Plan:

> If we move servlet initialization into Java code and get rid of web.xml, we may be able to start Solr in the same way both in tests and in real life!


I feel that this should be a top level goal, along with taking another look at EmbeddedSolrServer (perhaps it moves out of SolrJ?). Right now, I feel like we effectively have no testing of the configurations that go into the various xml files - can't speak for others, but it made me reluctant to make changes to them.


Mike

On Tue, Mar 24, 2020 at 6:36 PM Jan Høydahl <[hidden email]> wrote:
This SIP has changed name to SIP-6 and I have changed the email subject of this thread, please keep this new subject going forward.
New SIP link: https://cwiki.apache.org/confluence/display/SOLR/SIP-6+Solr+should+own+the+bootstrap+process

Jan

24. mar. 2020 kl. 22:48 skrev Uwe Schindler <[hidden email]>:

Of course it's a single jvm. It's only a replacement for start.jar. the mock-up is already there.

Uwe

Am March 24, 2020 9:45:33 PM UTC schrieb Bram Van Dam <[hidden email]>:
On 24/03/2020 16:01, Jan Høydahl wrote:
It has been proposed many times to make Solr a truly standalone app and that we should start Jetty from within Solr and not the other way around :)

Do you see this running in a single JVM, i.e. you run solr.jar and it
then starts an embedded Jetty server? Or would solr.jar exec a new Java
process and take it from there?

If it's all in one JVM, then the startup scripts probably won't be all
that different. Memory settings, OOM killers etc will still have to be
set up before using startup shellscripts and whatnot. If it's just a
small bootstrap jar which execs another JVM, a lot of the complicated
shell foo could probably be simplified.

- Bram
To unsubscribe, [hidden email]
For additional commands, [hidden email]


--
Uwe Schindler
Achterdiek 19, 28357 Bremen
https://www.thetaphi.de


Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] SIP-6 Solr should own the bootstrap process

Eric Pugh-4
Would being able to manage CORS settings be part of this?  

Now that Solr is ramping up it’s default security posture, there are many applications that used to work that don’t.  Today, changing Solr’s CORS settings is very cumbersome, requiring changing an embedded web.xml.

Making it easier to set your CORS type settings would be great.

Eric

On Mar 25, 2020, at 12:29 PM, Jan Høydahl <[hidden email]> wrote:

I agree. I changed the SIP page and now have four bullet points defined as in scope for the SIP:

The most important change in this SIP will be

  • Create a solr/bootstrap module with a solr.jar  to replace start.jar  as the entrypoint (java -jar solr.jar ...)
  • Hide jetty even more, i.e. rename jetty.port  → port , and perhaps clean up other Sysprops as well
  • Get rid of web.xml and instead build our contexts and paths from Java code, driven by Solr config
  • Be able to lock Solr's memory to prevent swapping (SOLR-14335)

Not sure how big of a job the web.xml thing is, but there is definitely a potential of unifying test and production code in this area and thus get much better test coverage of some of this.

However, removing web.xml is perhaps such a big change that we’ll have to target 9.0 while my initial goal of just creating a bootstrapper to lock memory could have been merged to 8.x as well?? Or do we consider web.xml an implementation detail for which we don’t need to provide backward compatibility?

I’m a bit back and forth with myself on this. Perhaps if we reduce scope to just the bootstrap module, mem locking and some sysprop renames, it can be done quickly and end up in 8.6 already, then increment from there?

Jan

25. mar. 2020 kl. 16:13 skrev Mike Drob <[hidden email]>:

I'm in favor of this approach mainly because of the line mentioned in the Test Plan:

> If we move servlet initialization into Java code and get rid of web.xml, we may be able to start Solr in the same way both in tests and in real life!


I feel that this should be a top level goal, along with taking another look at EmbeddedSolrServer (perhaps it moves out of SolrJ?). Right now, I feel like we effectively have no testing of the configurations that go into the various xml files - can't speak for others, but it made me reluctant to make changes to them.


Mike

On Tue, Mar 24, 2020 at 6:36 PM Jan Høydahl <[hidden email]> wrote:
This SIP has changed name to SIP-6 and I have changed the email subject of this thread, please keep this new subject going forward.
New SIP link: https://cwiki.apache.org/confluence/display/SOLR/SIP-6+Solr+should+own+the+bootstrap+process

Jan

24. mar. 2020 kl. 22:48 skrev Uwe Schindler <[hidden email]>:

Of course it's a single jvm. It's only a replacement for start.jar. the mock-up is already there.

Uwe

Am March 24, 2020 9:45:33 PM UTC schrieb Bram Van Dam <[hidden email]>:
On 24/03/2020 16:01, Jan Høydahl wrote:
It has been proposed many times to make Solr a truly standalone app and that we should start Jetty from within Solr and not the other way around :)

Do you see this running in a single JVM, i.e. you run solr.jar and it
then starts an embedded Jetty server? Or would solr.jar exec a new Java
process and take it from there?

If it's all in one JVM, then the startup scripts probably won't be all
that different. Memory settings, OOM killers etc will still have to be
set up before using startup shellscripts and whatnot. If it's just a
small bootstrap jar which execs another JVM, a lot of the complicated
shell foo could probably be simplified.

- Bram
To unsubscribe, [hidden email]
For additional commands, [hidden email]


--
Uwe Schindler
Achterdiek 19, 28357 Bremen
https://www.thetaphi.de



_______________________
Eric Pugh Founder & CEO | OpenSource Connections, LLC | 434.466.1467 | http://www.opensourceconnections.com | My Free/Busy  
This e-mail and all contents, including attachments, is considered to be Company Confidential unless explicitly stated otherwise, regardless of whether attachments are marked as such.

Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] SIP-6 Solr should own the bootstrap process

Mike Drob-3
In reply to this post by Jan Høydahl / Cominvent
I think we like to pretend that Jetty and web.xml are implementation details, but I'm not sure how good of a job we do.

Whether we need to provide backwards compatibility or not is the wrong question I think, because when we break compatibility we are telling a subset of our users that it will not be easy for them to upgrade. I don't know who out there has customizations in web.xml (maybe security filters is the most common?). I don't know what the migration path for those users would look like if they want to add stuff to an embedded jetty that we're running - do we need to provide system properties they can set? Do we need to give them a separate xml file that they can use similar syntax for?

I'm definitely against changing that for 8.6, and as much as I want this change for 9.0 I really want to see an alternative that makes sense proposed before we start tearing through all of this. I'll spend a bit more time thinking about this and see if I can come up with any ideas as well.

Mike

On Wed, Mar 25, 2020 at 11:29 AM Jan Høydahl <[hidden email]> wrote:
I agree. I changed the SIP page and now have four bullet points defined as in scope for the SIP:

The most important change in this SIP will be

  • Create a solr/bootstrap module with a solr.jar  to replace start.jar  as the entrypoint (java -jar solr.jar ...)
  • Hide jetty even more, i.e. rename jetty.port  → port , and perhaps clean up other Sysprops as well
  • Get rid of web.xml and instead build our contexts and paths from Java code, driven by Solr config
  • Be able to lock Solr's memory to prevent swapping (SOLR-14335)

Not sure how big of a job the web.xml thing is, but there is definitely a potential of unifying test and production code in this area and thus get much better test coverage of some of this.

However, removing web.xml is perhaps such a big change that we’ll have to target 9.0 while my initial goal of just creating a bootstrapper to lock memory could have been merged to 8.x as well?? Or do we consider web.xml an implementation detail for which we don’t need to provide backward compatibility?

I’m a bit back and forth with myself on this. Perhaps if we reduce scope to just the bootstrap module, mem locking and some sysprop renames, it can be done quickly and end up in 8.6 already, then increment from there?

Jan

25. mar. 2020 kl. 16:13 skrev Mike Drob <[hidden email]>:

I'm in favor of this approach mainly because of the line mentioned in the Test Plan:

> If we move servlet initialization into Java code and get rid of web.xml, we may be able to start Solr in the same way both in tests and in real life!


I feel that this should be a top level goal, along with taking another look at EmbeddedSolrServer (perhaps it moves out of SolrJ?). Right now, I feel like we effectively have no testing of the configurations that go into the various xml files - can't speak for others, but it made me reluctant to make changes to them.


Mike

On Tue, Mar 24, 2020 at 6:36 PM Jan Høydahl <[hidden email]> wrote:
This SIP has changed name to SIP-6 and I have changed the email subject of this thread, please keep this new subject going forward.
New SIP link: https://cwiki.apache.org/confluence/display/SOLR/SIP-6+Solr+should+own+the+bootstrap+process

Jan

24. mar. 2020 kl. 22:48 skrev Uwe Schindler <[hidden email]>:

Of course it's a single jvm. It's only a replacement for start.jar. the mock-up is already there.

Uwe

Am March 24, 2020 9:45:33 PM UTC schrieb Bram Van Dam <[hidden email]>:
On 24/03/2020 16:01, Jan Høydahl wrote:
It has been proposed many times to make Solr a truly standalone app and that we should start Jetty from within Solr and not the other way around :)

Do you see this running in a single JVM, i.e. you run solr.jar and it
then starts an embedded Jetty server? Or would solr.jar exec a new Java
process and take it from there?

If it's all in one JVM, then the startup scripts probably won't be all
that different. Memory settings, OOM killers etc will still have to be
set up before using startup shellscripts and whatnot. If it's just a
small bootstrap jar which execs another JVM, a lot of the complicated
shell foo could probably be simplified.

- Bram
To unsubscribe, [hidden email]
For additional commands, [hidden email]


--
Uwe Schindler
Achterdiek 19, 28357 Bremen
https://www.thetaphi.de


Reply | Threaded
Open this post in threaded view
|

RE: [DISCUSS] SIP-6 Solr should own the bootstrap process

Uwe Schindler
In reply to this post by Eric Pugh-4

Hi,

 

I think Solr should ship with default CORS Filter enabled (it’s part of Jetty). E.g. Elasticsearch also ships with a suitable CORS configuration by default. You can just change some details (e.g. restrict hostnames, default is “*”) or dis/enable it completely. We can also be much more selective which endpoints allow CORS at all, this limits security risks.

 

Uwe

 

-----

Uwe Schindler

Achterdiek 19, D-28357 Bremen

https://www.thetaphi.de

eMail: [hidden email]

 

From: Eric Pugh <[hidden email]>
Sent: Wednesday, March 25, 2020 5:34 PM
To: [hidden email]
Subject: Re: [DISCUSS] SIP-6 Solr should own the bootstrap process

 

Would being able to manage CORS settings be part of this?  

 

Now that Solr is ramping up it’s default security posture, there are many applications that used to work that don’t.  Today, changing Solr’s CORS settings is very cumbersome, requiring changing an embedded web.xml.

 

Making it easier to set your CORS type settings would be great.

 

Eric



On Mar 25, 2020, at 12:29 PM, Jan Høydahl <[hidden email]> wrote:

 

I agree. I changed the SIP page and now have four bullet points defined as in scope for the SIP:

 

The most important change in this SIP will be

·        Create a solr/bootstrap module with a solr.jar  to replace start.jar  as the entrypoint (java -jar solr.jar ...)

·        Hide jetty even more, i.e. rename jetty.port  → port , and perhaps clean up other Sysprops as well

·        Get rid of web.xml and instead build our contexts and paths from Java code, driven by Solr config

·        Be able to lock Solr's memory to prevent swapping (SOLR-14335)

 

Not sure how big of a job the web.xml thing is, but there is definitely a potential of unifying test and production code in this area and thus get much better test coverage of some of this.

 

However, removing web.xml is perhaps such a big change that we’ll have to target 9.0 while my initial goal of just creating a bootstrapper to lock memory could have been merged to 8.x as well?? Or do we consider web.xml an implementation detail for which we don’t need to provide backward compatibility?

 

I’m a bit back and forth with myself on this. Perhaps if we reduce scope to just the bootstrap module, mem locking and some sysprop renames, it can be done quickly and end up in 8.6 already, then increment from there?

 

Jan



25. mar. 2020 kl. 16:13 skrev Mike Drob <[hidden email]>:

 

I'm in favor of this approach mainly because of the line mentioned in the Test Plan:

> If we move servlet initialization into Java code and get rid of web.xml, we may be able to start Solr in the same way both in tests and in real life!

 

I feel that this should be a top level goal, along with taking another look at EmbeddedSolrServer (perhaps it moves out of SolrJ?). Right now, I feel like we effectively have no testing of the configurations that go into the various xml files - can't speak for others, but it made me reluctant to make changes to them.

 

 

Mike

 

On Tue, Mar 24, 2020 at 6:36 PM Jan Høydahl <[hidden email]> wrote:

This SIP has changed name to SIP-6 and I have changed the email subject of this thread, please keep this new subject going forward.

New SIP link: https://cwiki.apache.org/confluence/display/SOLR/SIP-6+Solr+should+own+the+bootstrap+process

 

Jan



24. mar. 2020 kl. 22:48 skrev Uwe Schindler <[hidden email]>:

 

Of course it's a single jvm. It's only a replacement for start.jar. the mock-up is already there.

Uwe

Am March 24, 2020 9:45:33 PM UTC schrieb Bram Van Dam <[hidden email]>:

On 24/03/2020 16:01, Jan Høydahl wrote:
It has been proposed many times to make Solr a truly standalone app and that we should start Jetty from within Solr and not the other way around :)

Do you see this running in a single JVM, i.e. you run solr.jar and it
then starts an embedded Jetty server? Or would solr.jar exec a new Java
process and take it from there?

If it's all in one JVM, then the startup scripts probably won't be all
that different. Memory settings, OOM killers etc will still have to be
set up before using startup shellscripts and whatnot. If it's just a
small bootstrap jar which execs another JVM, a lot of the complicated
shell foo could probably be simplified.

- Bram

To unsubscribe, [hidden email]
For additional commands, [hidden email]


--
Uwe Schindler
Achterdiek 19, 28357 Bremen
https://www.thetaphi.de

 

 

 

_______________________

Eric Pugh Founder & CEO | OpenSource Connections, LLC | 434.466.1467 | http://www.opensourceconnections.com | My Free/Busy  

This e-mail and all contents, including attachments, is considered to be Company Confidential unless explicitly stated otherwise, regardless of whether attachments are marked as such.

 

Reply | Threaded
Open this post in threaded view
|

RE: [DISCUSS] SIP-6 Solr should own the bootstrap process

Uwe Schindler
In reply to this post by Mike Drob-3

Hi,

 

I agree, this should be a change in 9.0 only.

 

In general we can start with web.xml still existing (the bootstrap code would use a Jetty “WebAppContext” initially). At a later stage we can replace the WebappContext by a ServletContextHandler (which is the superclass and offers no webapp anymore) and we can configure everything on our own.

 

Another idea would be to allow the end user to suply his own web.xml somewhere in a directory, where he can add extra filters if really needed. On bootstrap we setup our own WebappContext and tell it to load the user’s web.xml, but linking our own servlets and filters and assets would be done in our code. I don’t really like this approach, because this would disallow to replace jetty by async Netty in the future.

 

Uwe

 

-----

Uwe Schindler

Achterdiek 19, D-28357 Bremen

https://www.thetaphi.de

eMail: [hidden email]

 

From: Mike Drob <[hidden email]>
Sent: Wednesday, March 25, 2020 6:07 PM
To: [hidden email]
Subject: Re: [DISCUSS] SIP-6 Solr should own the bootstrap process

 

I think we like to pretend that Jetty and web.xml are implementation details, but I'm not sure how good of a job we do.

 

Whether we need to provide backwards compatibility or not is the wrong question I think, because when we break compatibility we are telling a subset of our users that it will not be easy for them to upgrade. I don't know who out there has customizations in web.xml (maybe security filters is the most common?). I don't know what the migration path for those users would look like if they want to add stuff to an embedded jetty that we're running - do we need to provide system properties they can set? Do we need to give them a separate xml file that they can use similar syntax for?

 

I'm definitely against changing that for 8.6, and as much as I want this change for 9.0 I really want to see an alternative that makes sense proposed before we start tearing through all of this. I'll spend a bit more time thinking about this and see if I can come up with any ideas as well.

 

Mike

 

On Wed, Mar 25, 2020 at 11:29 AM Jan Høydahl <[hidden email]> wrote:

I agree. I changed the SIP page and now have four bullet points defined as in scope for the SIP:

 

The most important change in this SIP will be

·        Create a solr/bootstrap module with a solr.jar  to replace start.jar  as the entrypoint (java -jar solr.jar ...)

·        Hide jetty even more, i.e. rename jetty.port  → port , and perhaps clean up other Sysprops as well

·        Get rid of web.xml and instead build our contexts and paths from Java code, driven by Solr config

·        Be able to lock Solr's memory to prevent swapping (SOLR-14335)

 

Not sure how big of a job the web.xml thing is, but there is definitely a potential of unifying test and production code in this area and thus get much better test coverage of some of this.

 

However, removing web.xml is perhaps such a big change that we’ll have to target 9.0 while my initial goal of just creating a bootstrapper to lock memory could have been merged to 8.x as well?? Or do we consider web.xml an implementation detail for which we don’t need to provide backward compatibility?

 

I’m a bit back and forth with myself on this. Perhaps if we reduce scope to just the bootstrap module, mem locking and some sysprop renames, it can be done quickly and end up in 8.6 already, then increment from there?

 

Jan



25. mar. 2020 kl. 16:13 skrev Mike Drob <[hidden email]>:

 

I'm in favor of this approach mainly because of the line mentioned in the Test Plan:

> If we move servlet initialization into Java code and get rid of web.xml, we may be able to start Solr in the same way both in tests and in real life!

 

I feel that this should be a top level goal, along with taking another look at EmbeddedSolrServer (perhaps it moves out of SolrJ?). Right now, I feel like we effectively have no testing of the configurations that go into the various xml files - can't speak for others, but it made me reluctant to make changes to them.

 

 

Mike

 

On Tue, Mar 24, 2020 at 6:36 PM Jan Høydahl <[hidden email]> wrote:

This SIP has changed name to SIP-6 and I have changed the email subject of this thread, please keep this new subject going forward.

New SIP link: https://cwiki.apache.org/confluence/display/SOLR/SIP-6+Solr+should+own+the+bootstrap+process

 

Jan



24. mar. 2020 kl. 22:48 skrev Uwe Schindler <[hidden email]>:

 

Of course it's a single jvm. It's only a replacement for start.jar. the mock-up is already there.

Uwe

Am March 24, 2020 9:45:33 PM UTC schrieb Bram Van Dam <[hidden email]>:

On 24/03/2020 16:01, Jan Høydahl wrote:
It has been proposed many times to make Solr a truly standalone app and that we should start Jetty from within Solr and not the other way around :)

Do you see this running in a single JVM, i.e. you run solr.jar and it
then starts an embedded Jetty server? Or would solr.jar exec a new Java
process and take it from there?

If it's all in one JVM, then the startup scripts probably won't be all
that different. Memory settings, OOM killers etc will still have to be
set up before using startup shellscripts and whatnot. If it's just a
small bootstrap jar which execs another JVM, a lot of the complicated
shell foo could probably be simplified.

- Bram

To unsubscribe, [hidden email]
For additional commands, [hidden email]


--
Uwe Schindler
Achterdiek 19, 28357 Bremen
https://www.thetaphi.de

 

 

Reply | Threaded
Open this post in threaded view
|

RE: [DISCUSS] SIP-6 Solr should own the bootstrap process

Uwe Schindler

We could use <https://www.eclipse.org/jetty/javadoc/9.4.14.v20181114/org/eclipse/jetty/webapp/WebAppContext.html#addOverrideDescriptor-java.lang.String-> to allow to override our config with a custom user-specified web.xml.

 

public void addOverrideDescriptor(java.lang.String overrideDescriptor)

The override descriptor is a web.xml format file that is applied to the context after the standard WEB-INF/web.xml

 

Uwe

 

-----

Uwe Schindler

Achterdiek 19, D-28357 Bremen

https://www.thetaphi.de

eMail: [hidden email]

 

From: Uwe Schindler <[hidden email]>
Sent: Wednesday, March 25, 2020 6:44 PM
To: [hidden email]
Subject: RE: [DISCUSS] SIP-6 Solr should own the bootstrap process

 

Hi,

 

I agree, this should be a change in 9.0 only.

 

In general we can start with web.xml still existing (the bootstrap code would use a Jetty “WebAppContext” initially). At a later stage we can replace the WebappContext by a ServletContextHandler (which is the superclass and offers no webapp anymore) and we can configure everything on our own.

 

Another idea would be to allow the end user to suply his own web.xml somewhere in a directory, where he can add extra filters if really needed. On bootstrap we setup our own WebappContext and tell it to load the user’s web.xml, but linking our own servlets and filters and assets would be done in our code. I don’t really like this approach, because this would disallow to replace jetty by async Netty in the future.

 

Uwe

 

-----

Uwe Schindler

Achterdiek 19, D-28357 Bremen

https://www.thetaphi.de

eMail: [hidden email]

 

From: Mike Drob <[hidden email]>
Sent: Wednesday, March 25, 2020 6:07 PM
To: [hidden email]
Subject: Re: [DISCUSS] SIP-6 Solr should own the bootstrap process

 

I think we like to pretend that Jetty and web.xml are implementation details, but I'm not sure how good of a job we do.

 

Whether we need to provide backwards compatibility or not is the wrong question I think, because when we break compatibility we are telling a subset of our users that it will not be easy for them to upgrade. I don't know who out there has customizations in web.xml (maybe security filters is the most common?). I don't know what the migration path for those users would look like if they want to add stuff to an embedded jetty that we're running - do we need to provide system properties they can set? Do we need to give them a separate xml file that they can use similar syntax for?

 

I'm definitely against changing that for 8.6, and as much as I want this change for 9.0 I really want to see an alternative that makes sense proposed before we start tearing through all of this. I'll spend a bit more time thinking about this and see if I can come up with any ideas as well.

 

Mike

 

On Wed, Mar 25, 2020 at 11:29 AM Jan Høydahl <[hidden email]> wrote:

I agree. I changed the SIP page and now have four bullet points defined as in scope for the SIP:

 

The most important change in this SIP will be

·        Create a solr/bootstrap module with a solr.jar  to replace start.jar  as the entrypoint (java -jar solr.jar ...)

·        Hide jetty even more, i.e. rename jetty.port  → port , and perhaps clean up other Sysprops as well

·        Get rid of web.xml and instead build our contexts and paths from Java code, driven by Solr config

·        Be able to lock Solr's memory to prevent swapping (SOLR-14335)

 

Not sure how big of a job the web.xml thing is, but there is definitely a potential of unifying test and production code in this area and thus get much better test coverage of some of this.

 

However, removing web.xml is perhaps such a big change that we’ll have to target 9.0 while my initial goal of just creating a bootstrapper to lock memory could have been merged to 8.x as well?? Or do we consider web.xml an implementation detail for which we don’t need to provide backward compatibility?

 

I’m a bit back and forth with myself on this. Perhaps if we reduce scope to just the bootstrap module, mem locking and some sysprop renames, it can be done quickly and end up in 8.6 already, then increment from there?

 

Jan

 

25. mar. 2020 kl. 16:13 skrev Mike Drob <[hidden email]>:

 

I'm in favor of this approach mainly because of the line mentioned in the Test Plan:

> If we move servlet initialization into Java code and get rid of web.xml, we may be able to start Solr in the same way both in tests and in real life!

 

I feel that this should be a top level goal, along with taking another look at EmbeddedSolrServer (perhaps it moves out of SolrJ?). Right now, I feel like we effectively have no testing of the configurations that go into the various xml files - can't speak for others, but it made me reluctant to make changes to them.

 

 

Mike

 

On Tue, Mar 24, 2020 at 6:36 PM Jan Høydahl <[hidden email]> wrote:

This SIP has changed name to SIP-6 and I have changed the email subject of this thread, please keep this new subject going forward.

New SIP link: https://cwiki.apache.org/confluence/display/SOLR/SIP-6+Solr+should+own+the+bootstrap+process

 

Jan

 

24. mar. 2020 kl. 22:48 skrev Uwe Schindler <[hidden email]>:

 

Of course it's a single jvm. It's only a replacement for start.jar. the mock-up is already there.

Uwe

Am March 24, 2020 9:45:33 PM UTC schrieb Bram Van Dam <[hidden email]>:

On 24/03/2020 16:01, Jan Høydahl wrote:
It has been proposed many times to make Solr a truly standalone app and that we should start Jetty from within Solr and not the other way around :)

Do you see this running in a single JVM, i.e. you run solr.jar and it
then starts an embedded Jetty server? Or would solr.jar exec a new Java
process and take it from there?

If it's all in one JVM, then the startup scripts probably won't be all
that different. Memory settings, OOM killers etc will still have to be
set up before using startup shellscripts and whatnot. If it's just a
small bootstrap jar which execs another JVM, a lot of the complicated
shell foo could probably be simplified.

- Bram

To unsubscribe, [hidden email]
For additional commands, [hidden email]


--
Uwe Schindler
Achterdiek 19, 28357 Bremen
https://www.thetaphi.de

 

 

Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] SIP-6 Solr should own the bootstrap process

Jan Høydahl / Cominvent
I agree Uwe. The scope of SIP-6 can be to let SolrBootstrap load existing web.xml through WebappContext, and spin off a new JIRA for the ServletContextHandler approach.

See the updated SIP-6 text which reflects this.

Jan

25. mar. 2020 kl. 18:50 skrev Uwe Schindler <[hidden email]>:

 
public void addOverrideDescriptor(java.lang.String overrideDescriptor)
The override descriptor is a web.xml format file that is applied to the context after the standard WEB-INF/web.xml
 
Uwe
 
-----
Uwe Schindler
Achterdiek 19, D-28357 Bremen
 
From: Uwe Schindler <[hidden email]> 
Sent: Wednesday, March 25, 2020 6:44 PM
To: [hidden email]
Subject: RE: [DISCUSS] SIP-6 Solr should own the bootstrap process
 
Hi,
 
I agree, this should be a change in 9.0 only.
 
In general we can start with web.xml still existing (the bootstrap code would use a Jetty “WebAppContext” initially). At a later stage we can replace the WebappContext by a ServletContextHandler (which is the superclass and offers no webapp anymore) and we can configure everything on our own.
 
Another idea would be to allow the end user to suply his own web.xml somewhere in a directory, where he can add extra filters if really needed. On bootstrap we setup our own WebappContext and tell it to load the user’s web.xml, but linking our own servlets and filters and assets would be done in our code. I don’t really like this approach, because this would disallow to replace jetty by async Netty in the future.
 
Uwe
 
-----
Uwe Schindler
Achterdiek 19, D-28357 Bremen
 
From: Mike Drob <[hidden email]> 
Sent: Wednesday, March 25, 2020 6:07 PM
To: [hidden email]
Subject: Re: [DISCUSS] SIP-6 Solr should own the bootstrap process
 
I think we like to pretend that Jetty and web.xml are implementation details, but I'm not sure how good of a job we do.
 
Whether we need to provide backwards compatibility or not is the wrong question I think, because when we break compatibility we are telling a subset of our users that it will not be easy for them to upgrade. I don't know who out there has customizations in web.xml (maybe security filters is the most common?). I don't know what the migration path for those users would look like if they want to add stuff to an embedded jetty that we're running - do we need to provide system properties they can set? Do we need to give them a separate xml file that they can use similar syntax for?
 
I'm definitely against changing that for 8.6, and as much as I want this change for 9.0 I really want to see an alternative that makes sense proposed before we start tearing through all of this. I'll spend a bit more time thinking about this and see if I can come up with any ideas as well.
 
Mike
 
On Wed, Mar 25, 2020 at 11:29 AM Jan Høydahl <[hidden email]> wrote:
I agree. I changed the SIP page and now have four bullet points defined as in scope for the SIP:
 

The most important change in this SIP will be

·        Create a solr/bootstrap module with a solr.jar  to replace start.jar  as the entrypoint (java -jar solr.jar ...)
·        Hide jetty even more, i.e. rename jetty.port  → port , and perhaps clean up other Sysprops as well
·        Get rid of web.xml and instead build our contexts and paths from Java code, driven by Solr config
·        Be able to lock Solr's memory to prevent swapping (SOLR-14335)
 
Not sure how big of a job the web.xml thing is, but there is definitely a potential of unifying test and production code in this area and thus get much better test coverage of some of this.
 
However, removing web.xml is perhaps such a big change that we’ll have to target 9.0 while my initial goal of just creating a bootstrapper to lock memory could have been merged to 8.x as well?? Or do we consider web.xml an implementation detail for which we don’t need to provide backward compatibility?
 
I’m a bit back and forth with myself on this. Perhaps if we reduce scope to just the bootstrap module, mem locking and some sysprop renames, it can be done quickly and end up in 8.6 already, then increment from there?
 
Jan

 

25. mar. 2020 kl. 16:13 skrev Mike Drob <[hidden email]>:
 
I'm in favor of this approach mainly because of the line mentioned in the Test Plan:

> If we move servlet initialization into Java code and get rid of web.xml, we may be able to start Solr in the same way both in tests and in real life!

 
I feel that this should be a top level goal, along with taking another look at EmbeddedSolrServer (perhaps it moves out of SolrJ?). Right now, I feel like we effectively have no testing of the configurations that go into the various xml files - can't speak for others, but it made me reluctant to make changes to them.
 
 
Mike
 
On Tue, Mar 24, 2020 at 6:36 PM Jan Høydahl <[hidden email]> wrote:
This SIP has changed name to SIP-6 and I have changed the email subject of this thread, please keep this new subject going forward.
 
Jan

 

24. mar. 2020 kl. 22:48 skrev Uwe Schindler <[hidden email]>:
 

Of course it's a single jvm. It's only a replacement for start.jar. the mock-up is already there.

Uwe

Am March 24, 2020 9:45:33 PM UTC schrieb Bram Van Dam <[hidden email]>:
On 24/03/2020 16:01, Jan Høydahl wrote:
It has been proposed many times to make Solr a truly standalone app and that we should start Jetty from within Solr and not the other way around :)

Do you see this running in a single JVM, i.e. you run solr.jar and it
then starts an embedded Jetty server? Or would solr.jar exec a new Java
process and take it from there?

If it's all in one JVM, then the startup scripts probably won't be all
that different. Memory settings, OOM killers etc will still have to be
set up before using startup shellscripts and whatnot. If it's just a
small bootstrap jar which execs another JVM, a lot of the complicated
shell foo could probably be simplified.

- Bram

To unsubscribe, [hidden email]
For additional commands, [hidden email]

--
Uwe Schindler
Achterdiek 19, 28357 Bremen
https://www.thetaphi.de