The Lucene Solr Gradle Build Game plan

classic Classic list List threaded Threaded
52 messages Options
123
Reply | Threaded
Open this post in threaded view
|

The Lucene Solr Gradle Build Game plan

Mark Miller-3
https://issues.apache.org/jira/browse/SOLR-13452
Update the lucene-solr build from Ivy+Ant+Maven (shadow build) to Gradle.

Hey all.

Here is my Lucene and Solr 'move to Gradle' plans.

* I plan on trying to commit the Gradle build by 9/30
* In the meantime I'll do a bit more work on packaging, publishing, and dependencies.
* It would be great if others could start digging in a bit as well.
* I plan to try and sidecar Gradle to begin with. Once that happens, I really need others to start digging in as we have a lot to do (switch ci and other tools to the new build, figure out new release docs and issues, finish the Solr documentation module (I'll try and get to that earlier than later), and generally make things pleasant across as many dev envs as we can.
* Beyond that, there will likely be a longer tail of smaller issues or missing things even after we make the switch.
* I'll put some effort into the side car experiment for another month or two perhaps (maybe not much in November)
* If things are not looking like we will be able to flip the switch, I'll look at pulling out Gradle.

- Mark
Reply | Threaded
Open this post in threaded view
|

Re: The Lucene Solr Gradle Build Game plan

Mark Miller-3
I've started to put together a little guide to help people ramp up here: https://cwiki.apache.org/confluence/display/SOLR/Intro+to+the+Gradle+build

On Sat, Sep 14, 2019 at 9:09 PM Mark Miller <[hidden email]> wrote:
https://issues.apache.org/jira/browse/SOLR-13452
Update the lucene-solr build from Ivy+Ant+Maven (shadow build) to Gradle.

Hey all.

Here is my Lucene and Solr 'move to Gradle' plans.

* I plan on trying to commit the Gradle build by 9/30
* In the meantime I'll do a bit more work on packaging, publishing, and dependencies.
* It would be great if others could start digging in a bit as well.
* I plan to try and sidecar Gradle to begin with. Once that happens, I really need others to start digging in as we have a lot to do (switch ci and other tools to the new build, figure out new release docs and issues, finish the Solr documentation module (I'll try and get to that earlier than later), and generally make things pleasant across as many dev envs as we can.
* Beyond that, there will likely be a longer tail of smaller issues or missing things even after we make the switch.
* I'll put some effort into the side car experiment for another month or two perhaps (maybe not much in November)
* If things are not looking like we will be able to flip the switch, I'll look at pulling out Gradle.

- Mark


--
Reply | Threaded
Open this post in threaded view
|

Re: The Lucene Solr Gradle Build Game plan

Mark Miller-3
Okay, I've tried to spread that link a little on social media as well.

Please do some experimentation. Especially those of you that use or know more esoteric things about the build. The basics are pretty solid, most have been working for months now, it's the corners and crannies that I'm more concerned about. You could have built and run tests a few months ago and said, nice, it's working, but then I've done 10,000 things since then that were necessary. You may easily not realize things are a problem until you really dig into something.

The more that devs can start trying out Gradle now, the less overlap of the two build systems we will need.

The idea of the overlap is that it will almost force many of us to start trying things out - at which point we will start to understand any key things that are missing, key problems or bugs, etc. If I just make the switch and let the cards fall were they may, those cards are almost certainly going to be very disruptive and annoying to a lot of people.

It will also give us a chance to start rolling CI and other tools over to using the Gradle build while ant+ivy are still available and in charge. I don't think it's a great idea to try and do this all at one moment, that will be difficult to coordinate and be more disruptive to less interested devs. We would like to keep the dual build situation contained and short though. That's why I have a plan to pull out if we don't make enough progress by a month or twos time (closer to a month). 

If we have enough experimentation ahead of time, the overlap can be very short - we start moving things like jenkins jobs over and when we are done and confident the world is not ending, we make some adjustments so that gradle owns the build (eg the dependency tree) and then remove ant+ivy+maven.

Some things like the smoke tester and what not *can* come shortly after we switch I think - we have until the 9 release to truly get everything in order (keeping in mind that we are still developing so some things are still critical). But we need CI and other basics all moved over when we flip the ownership switch.

- Mark

On Sun, Sep 15, 2019 at 12:28 PM Mark Miller <[hidden email]> wrote:
I've started to put together a little guide to help people ramp up here: https://cwiki.apache.org/confluence/display/SOLR/Intro+to+the+Gradle+build

On Sat, Sep 14, 2019 at 9:09 PM Mark Miller <[hidden email]> wrote:
https://issues.apache.org/jira/browse/SOLR-13452
Update the lucene-solr build from Ivy+Ant+Maven (shadow build) to Gradle.

Hey all.

Here is my Lucene and Solr 'move to Gradle' plans.

* I plan on trying to commit the Gradle build by 9/30
* In the meantime I'll do a bit more work on packaging, publishing, and dependencies.
* It would be great if others could start digging in a bit as well.
* I plan to try and sidecar Gradle to begin with. Once that happens, I really need others to start digging in as we have a lot to do (switch ci and other tools to the new build, figure out new release docs and issues, finish the Solr documentation module (I'll try and get to that earlier than later), and generally make things pleasant across as many dev envs as we can.
* Beyond that, there will likely be a longer tail of smaller issues or missing things even after we make the switch.
* I'll put some effort into the side car experiment for another month or two perhaps (maybe not much in November)
* If things are not looking like we will be able to flip the switch, I'll look at pulling out Gradle.

- Mark


--


--
Reply | Threaded
Open this post in threaded view
|

Re: The Lucene Solr Gradle Build Game plan

Gus Heck
Not sure if you heard, but about a half a dozen folks tried it out on macs and one on windows at the hack day on Tuesday before Activate. It caused some scrambling for sharing of power bricks (a single run of the tests eats 70% of a fully charged 2018 macbook pro battery in 45 min), but the good news is it only failed on well known flaky tests and on the one windows machine and that in the PDF building for the ref guide (and there was that small bit with the error message and the AtomicBoolean that I fixed). I've heard some opinions that maybe we don't need the PDF version in the future, The one bit of feedback that came out of it was it would be nice to have a task that tweaked the configs things to not peg the processor quite so efficiently :). Certainly we do want the mode for full utilization to be available, but a background mode would be good too in case folks need to do other work.

-Gus

On Sun, Sep 15, 2019 at 3:24 PM Mark Miller <[hidden email]> wrote:
Okay, I've tried to spread that link a little on social media as well.

Please do some experimentation. Especially those of you that use or know more esoteric things about the build. The basics are pretty solid, most have been working for months now, it's the corners and crannies that I'm more concerned about. You could have built and run tests a few months ago and said, nice, it's working, but then I've done 10,000 things since then that were necessary. You may easily not realize things are a problem until you really dig into something.

The more that devs can start trying out Gradle now, the less overlap of the two build systems we will need.

The idea of the overlap is that it will almost force many of us to start trying things out - at which point we will start to understand any key things that are missing, key problems or bugs, etc. If I just make the switch and let the cards fall were they may, those cards are almost certainly going to be very disruptive and annoying to a lot of people.

It will also give us a chance to start rolling CI and other tools over to using the Gradle build while ant+ivy are still available and in charge. I don't think it's a great idea to try and do this all at one moment, that will be difficult to coordinate and be more disruptive to less interested devs. We would like to keep the dual build situation contained and short though. That's why I have a plan to pull out if we don't make enough progress by a month or twos time (closer to a month). 

If we have enough experimentation ahead of time, the overlap can be very short - we start moving things like jenkins jobs over and when we are done and confident the world is not ending, we make some adjustments so that gradle owns the build (eg the dependency tree) and then remove ant+ivy+maven.

Some things like the smoke tester and what not *can* come shortly after we switch I think - we have until the 9 release to truly get everything in order (keeping in mind that we are still developing so some things are still critical). But we need CI and other basics all moved over when we flip the ownership switch.

- Mark

On Sun, Sep 15, 2019 at 12:28 PM Mark Miller <[hidden email]> wrote:
I've started to put together a little guide to help people ramp up here: https://cwiki.apache.org/confluence/display/SOLR/Intro+to+the+Gradle+build

On Sat, Sep 14, 2019 at 9:09 PM Mark Miller <[hidden email]> wrote:
https://issues.apache.org/jira/browse/SOLR-13452
Update the lucene-solr build from Ivy+Ant+Maven (shadow build) to Gradle.

Hey all.

Here is my Lucene and Solr 'move to Gradle' plans.

* I plan on trying to commit the Gradle build by 9/30
* In the meantime I'll do a bit more work on packaging, publishing, and dependencies.
* It would be great if others could start digging in a bit as well.
* I plan to try and sidecar Gradle to begin with. Once that happens, I really need others to start digging in as we have a lot to do (switch ci and other tools to the new build, figure out new release docs and issues, finish the Solr documentation module (I'll try and get to that earlier than later), and generally make things pleasant across as many dev envs as we can.
* Beyond that, there will likely be a longer tail of smaller issues or missing things even after we make the switch.
* I'll put some effort into the side car experiment for another month or two perhaps (maybe not much in November)
* If things are not looking like we will be able to flip the switch, I'll look at pulling out Gradle.

- Mark


--


--


--
Reply | Threaded
Open this post in threaded view
|

Re: The Lucene Solr Gradle Build Game plan

Mark Miller-3
I've added more about that here: https://cwiki.apache.org/confluence/display/SOLR/Intro+to+the+Gradle+build

It's configurable, but difficult for us to choose a default based on cores as far as I can tell and gradles default, which is based on cores detected, is too high, especially because of hyperthreading.

Probably the best we can do is default it to a hard 2 or 4 and let people raise it depending on what wait you want to error. In both cases the majority of people will want to change it.

On Sun, Sep 15, 2019 at 3:39 PM Gus Heck <[hidden email]> wrote:
Not sure if you heard, but about a half a dozen folks tried it out on macs and one on windows at the hack day on Tuesday before Activate. It caused some scrambling for sharing of power bricks (a single run of the tests eats 70% of a fully charged 2018 macbook pro battery in 45 min), but the good news is it only failed on well known flaky tests and on the one windows machine and that in the PDF building for the ref guide (and there was that small bit with the error message and the AtomicBoolean that I fixed). I've heard some opinions that maybe we don't need the PDF version in the future, The one bit of feedback that came out of it was it would be nice to have a task that tweaked the configs things to not peg the processor quite so efficiently :). Certainly we do want the mode for full utilization to be available, but a background mode would be good too in case folks need to do other work.

-Gus

On Sun, Sep 15, 2019 at 3:24 PM Mark Miller <[hidden email]> wrote:
Okay, I've tried to spread that link a little on social media as well.

Please do some experimentation. Especially those of you that use or know more esoteric things about the build. The basics are pretty solid, most have been working for months now, it's the corners and crannies that I'm more concerned about. You could have built and run tests a few months ago and said, nice, it's working, but then I've done 10,000 things since then that were necessary. You may easily not realize things are a problem until you really dig into something.

The more that devs can start trying out Gradle now, the less overlap of the two build systems we will need.

The idea of the overlap is that it will almost force many of us to start trying things out - at which point we will start to understand any key things that are missing, key problems or bugs, etc. If I just make the switch and let the cards fall were they may, those cards are almost certainly going to be very disruptive and annoying to a lot of people.

It will also give us a chance to start rolling CI and other tools over to using the Gradle build while ant+ivy are still available and in charge. I don't think it's a great idea to try and do this all at one moment, that will be difficult to coordinate and be more disruptive to less interested devs. We would like to keep the dual build situation contained and short though. That's why I have a plan to pull out if we don't make enough progress by a month or twos time (closer to a month). 

If we have enough experimentation ahead of time, the overlap can be very short - we start moving things like jenkins jobs over and when we are done and confident the world is not ending, we make some adjustments so that gradle owns the build (eg the dependency tree) and then remove ant+ivy+maven.

Some things like the smoke tester and what not *can* come shortly after we switch I think - we have until the 9 release to truly get everything in order (keeping in mind that we are still developing so some things are still critical). But we need CI and other basics all moved over when we flip the ownership switch.

- Mark

On Sun, Sep 15, 2019 at 12:28 PM Mark Miller <[hidden email]> wrote:
I've started to put together a little guide to help people ramp up here: https://cwiki.apache.org/confluence/display/SOLR/Intro+to+the+Gradle+build

On Sat, Sep 14, 2019 at 9:09 PM Mark Miller <[hidden email]> wrote:
https://issues.apache.org/jira/browse/SOLR-13452
Update the lucene-solr build from Ivy+Ant+Maven (shadow build) to Gradle.

Hey all.

Here is my Lucene and Solr 'move to Gradle' plans.

* I plan on trying to commit the Gradle build by 9/30
* In the meantime I'll do a bit more work on packaging, publishing, and dependencies.
* It would be great if others could start digging in a bit as well.
* I plan to try and sidecar Gradle to begin with. Once that happens, I really need others to start digging in as we have a lot to do (switch ci and other tools to the new build, figure out new release docs and issues, finish the Solr documentation module (I'll try and get to that earlier than later), and generally make things pleasant across as many dev envs as we can.
* Beyond that, there will likely be a longer tail of smaller issues or missing things even after we make the switch.
* I'll put some effort into the side car experiment for another month or two perhaps (maybe not much in November)
* If things are not looking like we will be able to flip the switch, I'll look at pulling out Gradle.

- Mark


--


--


--


--
Reply | Threaded
Open this post in threaded view
|

Re: The Lucene Solr Gradle Build Game plan

Mark Miller-3
I'll just detail it out here as well:

You can configure parallelism in ~/.gradle/gradle.properties

org.gradle.workers.max=2
tests_jvms=5

org.gradle.workers.max is controlled by gradle and defaults to the number of cores detected - I wish I could change to divided by 2.
test_jvms is controlled by us and defaults to the number of cores detected / 2.

org.gradle.workers.max  controls the total number of jvms that will be run in parallel - for tasks or tests or whatever gradle is doing.
test_jvms controls how many parallel jvms a module will use for tests, but that is also limited by org.gradle.workers.max

You should try setting both to cores / 2 and work down from there if needed.

When running tests across multiple modules, org.gradle.workers.max is the actual limit for test jvms spun up because each module is only limited to test_jvms, but ALL tasks are limited to org.gradle.workers.max and module tasks are run in parallel by default.

By setting them the same, we get similar behavior whether we run tests from the root directory of the project (all tests) or from a single module (say solr-core).

On Sun, Sep 15, 2019 at 5:03 PM Mark Miller <[hidden email]> wrote:
I've added more about that here: https://cwiki.apache.org/confluence/display/SOLR/Intro+to+the+Gradle+build

It's configurable, but difficult for us to choose a default based on cores as far as I can tell and gradles default, which is based on cores detected, is too high, especially because of hyperthreading.

Probably the best we can do is default it to a hard 2 or 4 and let people raise it depending on what wait you want to error. In both cases the majority of people will want to change it.

On Sun, Sep 15, 2019 at 3:39 PM Gus Heck <[hidden email]> wrote:
Not sure if you heard, but about a half a dozen folks tried it out on macs and one on windows at the hack day on Tuesday before Activate. It caused some scrambling for sharing of power bricks (a single run of the tests eats 70% of a fully charged 2018 macbook pro battery in 45 min), but the good news is it only failed on well known flaky tests and on the one windows machine and that in the PDF building for the ref guide (and there was that small bit with the error message and the AtomicBoolean that I fixed). I've heard some opinions that maybe we don't need the PDF version in the future, The one bit of feedback that came out of it was it would be nice to have a task that tweaked the configs things to not peg the processor quite so efficiently :). Certainly we do want the mode for full utilization to be available, but a background mode would be good too in case folks need to do other work.

-Gus

On Sun, Sep 15, 2019 at 3:24 PM Mark Miller <[hidden email]> wrote:
Okay, I've tried to spread that link a little on social media as well.

Please do some experimentation. Especially those of you that use or know more esoteric things about the build. The basics are pretty solid, most have been working for months now, it's the corners and crannies that I'm more concerned about. You could have built and run tests a few months ago and said, nice, it's working, but then I've done 10,000 things since then that were necessary. You may easily not realize things are a problem until you really dig into something.

The more that devs can start trying out Gradle now, the less overlap of the two build systems we will need.

The idea of the overlap is that it will almost force many of us to start trying things out - at which point we will start to understand any key things that are missing, key problems or bugs, etc. If I just make the switch and let the cards fall were they may, those cards are almost certainly going to be very disruptive and annoying to a lot of people.

It will also give us a chance to start rolling CI and other tools over to using the Gradle build while ant+ivy are still available and in charge. I don't think it's a great idea to try and do this all at one moment, that will be difficult to coordinate and be more disruptive to less interested devs. We would like to keep the dual build situation contained and short though. That's why I have a plan to pull out if we don't make enough progress by a month or twos time (closer to a month). 

If we have enough experimentation ahead of time, the overlap can be very short - we start moving things like jenkins jobs over and when we are done and confident the world is not ending, we make some adjustments so that gradle owns the build (eg the dependency tree) and then remove ant+ivy+maven.

Some things like the smoke tester and what not *can* come shortly after we switch I think - we have until the 9 release to truly get everything in order (keeping in mind that we are still developing so some things are still critical). But we need CI and other basics all moved over when we flip the ownership switch.

- Mark

On Sun, Sep 15, 2019 at 12:28 PM Mark Miller <[hidden email]> wrote:
I've started to put together a little guide to help people ramp up here: https://cwiki.apache.org/confluence/display/SOLR/Intro+to+the+Gradle+build

On Sat, Sep 14, 2019 at 9:09 PM Mark Miller <[hidden email]> wrote:
https://issues.apache.org/jira/browse/SOLR-13452
Update the lucene-solr build from Ivy+Ant+Maven (shadow build) to Gradle.

Hey all.

Here is my Lucene and Solr 'move to Gradle' plans.

* I plan on trying to commit the Gradle build by 9/30
* In the meantime I'll do a bit more work on packaging, publishing, and dependencies.
* It would be great if others could start digging in a bit as well.
* I plan to try and sidecar Gradle to begin with. Once that happens, I really need others to start digging in as we have a lot to do (switch ci and other tools to the new build, figure out new release docs and issues, finish the Solr documentation module (I'll try and get to that earlier than later), and generally make things pleasant across as many dev envs as we can.
* Beyond that, there will likely be a longer tail of smaller issues or missing things even after we make the switch.
* I'll put some effort into the side car experiment for another month or two perhaps (maybe not much in November)
* If things are not looking like we will be able to flip the switch, I'll look at pulling out Gradle.

- Mark


--


--


--


--


--
Reply | Threaded
Open this post in threaded view
|

Re: The Lucene Solr Gradle Build Game plan

Mark Miller-3
By the way, I did hear about the hack day and some Gradle testing, which is great, and very useful. Totally needed, but we also need a bit of a more deep core developer view of things vs the old build as well. The type of stuff that’s much harder to tease out than verifying all the new build targets and such. Of course a lot of that can come after we switch, but I have a sneaky feeling some core devs will have deep opinions about certain things. 

Ill add a new task for default config setup. 

Mark

On Sun, Sep 15, 2019 at 5:12 PM Mark Miller <[hidden email]> wrote:
I'll just detail it out here as well:

You can configure parallelism in ~/.gradle/gradle.properties

org.gradle.workers.max=2
tests_jvms=5

org.gradle.workers.max is controlled by gradle and defaults to the number of cores detected - I wish I could change to divided by 2.
test_jvms is controlled by us and defaults to the number of cores detected / 2.

org.gradle.workers.max  controls the total number of jvms that will be run in parallel - for tasks or tests or whatever gradle is doing.
test_jvms controls how many parallel jvms a module will use for tests, but that is also limited by org.gradle.workers.max

You should try setting both to cores / 2 and work down from there if needed.

When running tests across multiple modules, org.gradle.workers.max is the actual limit for test jvms spun up because each module is only limited to test_jvms, but ALL tasks are limited to org.gradle.workers.max and module tasks are run in parallel by default.

By setting them the same, we get similar behavior whether we run tests from the root directory of the project (all tests) or from a single module (say solr-core).

On Sun, Sep 15, 2019 at 5:03 PM Mark Miller <[hidden email]> wrote:
I've added more about that here: https://cwiki.apache.org/confluence/display/SOLR/Intro+to+the+Gradle+build

It's configurable, but difficult for us to choose a default based on cores as far as I can tell and gradles default, which is based on cores detected, is too high, especially because of hyperthreading.

Probably the best we can do is default it to a hard 2 or 4 and let people raise it depending on what wait you want to error. In both cases the majority of people will want to change it.

On Sun, Sep 15, 2019 at 3:39 PM Gus Heck <[hidden email]> wrote:
Not sure if you heard, but about a half a dozen folks tried it out on macs and one on windows at the hack day on Tuesday before Activate. It caused some scrambling for sharing of power bricks (a single run of the tests eats 70% of a fully charged 2018 macbook pro battery in 45 min), but the good news is it only failed on well known flaky tests and on the one windows machine and that in the PDF building for the ref guide (and there was that small bit with the error message and the AtomicBoolean that I fixed). I've heard some opinions that maybe we don't need the PDF version in the future, The one bit of feedback that came out of it was it would be nice to have a task that tweaked the configs things to not peg the processor quite so efficiently :). Certainly we do want the mode for full utilization to be available, but a background mode would be good too in case folks need to do other work.

-Gus

On Sun, Sep 15, 2019 at 3:24 PM Mark Miller <[hidden email]> wrote:
Okay, I've tried to spread that link a little on social media as well.

Please do some experimentation. Especially those of you that use or know more esoteric things about the build. The basics are pretty solid, most have been working for months now, it's the corners and crannies that I'm more concerned about. You could have built and run tests a few months ago and said, nice, it's working, but then I've done 10,000 things since then that were necessary. You may easily not realize things are a problem until you really dig into something.

The more that devs can start trying out Gradle now, the less overlap of the two build systems we will need.

The idea of the overlap is that it will almost force many of us to start trying things out - at which point we will start to understand any key things that are missing, key problems or bugs, etc. If I just make the switch and let the cards fall were they may, those cards are almost certainly going to be very disruptive and annoying to a lot of people.

It will also give us a chance to start rolling CI and other tools over to using the Gradle build while ant+ivy are still available and in charge. I don't think it's a great idea to try and do this all at one moment, that will be difficult to coordinate and be more disruptive to less interested devs. We would like to keep the dual build situation contained and short though. That's why I have a plan to pull out if we don't make enough progress by a month or twos time (closer to a month). 

If we have enough experimentation ahead of time, the overlap can be very short - we start moving things like jenkins jobs over and when we are done and confident the world is not ending, we make some adjustments so that gradle owns the build (eg the dependency tree) and then remove ant+ivy+maven.

Some things like the smoke tester and what not *can* come shortly after we switch I think - we have until the 9 release to truly get everything in order (keeping in mind that we are still developing so some things are still critical). But we need CI and other basics all moved over when we flip the ownership switch.

- Mark

On Sun, Sep 15, 2019 at 12:28 PM Mark Miller <[hidden email]> wrote:
I've started to put together a little guide to help people ramp up here: https://cwiki.apache.org/confluence/display/SOLR/Intro+to+the+Gradle+build

On Sat, Sep 14, 2019 at 9:09 PM Mark Miller <[hidden email]> wrote:
https://issues.apache.org/jira/browse/SOLR-13452
Update the lucene-solr build from Ivy+Ant+Maven (shadow build) to Gradle.

Hey all.

Here is my Lucene and Solr 'move to Gradle' plans.

* I plan on trying to commit the Gradle build by 9/30
* In the meantime I'll do a bit more work on packaging, publishing, and dependencies.
* It would be great if others could start digging in a bit as well.
* I plan to try and sidecar Gradle to begin with. Once that happens, I really need others to start digging in as we have a lot to do (switch ci and other tools to the new build, figure out new release docs and issues, finish the Solr documentation module (I'll try and get to that earlier than later), and generally make things pleasant across as many dev envs as we can.
* Beyond that, there will likely be a longer tail of smaller issues or missing things even after we make the switch.
* I'll put some effort into the side car experiment for another month or two perhaps (maybe not much in November)
* If things are not looking like we will be able to flip the switch, I'll look at pulling out Gradle.

- Mark


--


--


--


--


--
--
Reply | Threaded
Open this post in threaded view
|

Re: The Lucene Solr Gradle Build Game plan

Mark Miller-3
Okay, there is now a task called defaultUserConfig

You can do gw defaultUserConfig to set some recommended settings in ~/.gradle/gradle.properties. By default they should be more relaxed, but we can tune as needed.

For better build performance but also more resource usage you can do:

gw defaultUserConfig --style=aggressive

Mark

On Sun, Sep 15, 2019 at 5:50 PM Mark Miller <[hidden email]> wrote:
By the way, I did hear about the hack day and some Gradle testing, which is great, and very useful. Totally needed, but we also need a bit of a more deep core developer view of things vs the old build as well. The type of stuff that’s much harder to tease out than verifying all the new build targets and such. Of course a lot of that can come after we switch, but I have a sneaky feeling some core devs will have deep opinions about certain things. 

Ill add a new task for default config setup. 

Mark

On Sun, Sep 15, 2019 at 5:12 PM Mark Miller <[hidden email]> wrote:
I'll just detail it out here as well:

You can configure parallelism in ~/.gradle/gradle.properties

org.gradle.workers.max=2
tests_jvms=5

org.gradle.workers.max is controlled by gradle and defaults to the number of cores detected - I wish I could change to divided by 2.
test_jvms is controlled by us and defaults to the number of cores detected / 2.

org.gradle.workers.max  controls the total number of jvms that will be run in parallel - for tasks or tests or whatever gradle is doing.
test_jvms controls how many parallel jvms a module will use for tests, but that is also limited by org.gradle.workers.max

You should try setting both to cores / 2 and work down from there if needed.

When running tests across multiple modules, org.gradle.workers.max is the actual limit for test jvms spun up because each module is only limited to test_jvms, but ALL tasks are limited to org.gradle.workers.max and module tasks are run in parallel by default.

By setting them the same, we get similar behavior whether we run tests from the root directory of the project (all tests) or from a single module (say solr-core).

On Sun, Sep 15, 2019 at 5:03 PM Mark Miller <[hidden email]> wrote:
I've added more about that here: https://cwiki.apache.org/confluence/display/SOLR/Intro+to+the+Gradle+build

It's configurable, but difficult for us to choose a default based on cores as far as I can tell and gradles default, which is based on cores detected, is too high, especially because of hyperthreading.

Probably the best we can do is default it to a hard 2 or 4 and let people raise it depending on what wait you want to error. In both cases the majority of people will want to change it.

On Sun, Sep 15, 2019 at 3:39 PM Gus Heck <[hidden email]> wrote:
Not sure if you heard, but about a half a dozen folks tried it out on macs and one on windows at the hack day on Tuesday before Activate. It caused some scrambling for sharing of power bricks (a single run of the tests eats 70% of a fully charged 2018 macbook pro battery in 45 min), but the good news is it only failed on well known flaky tests and on the one windows machine and that in the PDF building for the ref guide (and there was that small bit with the error message and the AtomicBoolean that I fixed). I've heard some opinions that maybe we don't need the PDF version in the future, The one bit of feedback that came out of it was it would be nice to have a task that tweaked the configs things to not peg the processor quite so efficiently :). Certainly we do want the mode for full utilization to be available, but a background mode would be good too in case folks need to do other work.

-Gus

On Sun, Sep 15, 2019 at 3:24 PM Mark Miller <[hidden email]> wrote:
Okay, I've tried to spread that link a little on social media as well.

Please do some experimentation. Especially those of you that use or know more esoteric things about the build. The basics are pretty solid, most have been working for months now, it's the corners and crannies that I'm more concerned about. You could have built and run tests a few months ago and said, nice, it's working, but then I've done 10,000 things since then that were necessary. You may easily not realize things are a problem until you really dig into something.

The more that devs can start trying out Gradle now, the less overlap of the two build systems we will need.

The idea of the overlap is that it will almost force many of us to start trying things out - at which point we will start to understand any key things that are missing, key problems or bugs, etc. If I just make the switch and let the cards fall were they may, those cards are almost certainly going to be very disruptive and annoying to a lot of people.

It will also give us a chance to start rolling CI and other tools over to using the Gradle build while ant+ivy are still available and in charge. I don't think it's a great idea to try and do this all at one moment, that will be difficult to coordinate and be more disruptive to less interested devs. We would like to keep the dual build situation contained and short though. That's why I have a plan to pull out if we don't make enough progress by a month or twos time (closer to a month). 

If we have enough experimentation ahead of time, the overlap can be very short - we start moving things like jenkins jobs over and when we are done and confident the world is not ending, we make some adjustments so that gradle owns the build (eg the dependency tree) and then remove ant+ivy+maven.

Some things like the smoke tester and what not *can* come shortly after we switch I think - we have until the 9 release to truly get everything in order (keeping in mind that we are still developing so some things are still critical). But we need CI and other basics all moved over when we flip the ownership switch.

- Mark

On Sun, Sep 15, 2019 at 12:28 PM Mark Miller <[hidden email]> wrote:
I've started to put together a little guide to help people ramp up here: https://cwiki.apache.org/confluence/display/SOLR/Intro+to+the+Gradle+build

On Sat, Sep 14, 2019 at 9:09 PM Mark Miller <[hidden email]> wrote:
https://issues.apache.org/jira/browse/SOLR-13452
Update the lucene-solr build from Ivy+Ant+Maven (shadow build) to Gradle.

Hey all.

Here is my Lucene and Solr 'move to Gradle' plans.

* I plan on trying to commit the Gradle build by 9/30
* In the meantime I'll do a bit more work on packaging, publishing, and dependencies.
* It would be great if others could start digging in a bit as well.
* I plan to try and sidecar Gradle to begin with. Once that happens, I really need others to start digging in as we have a lot to do (switch ci and other tools to the new build, figure out new release docs and issues, finish the Solr documentation module (I'll try and get to that earlier than later), and generally make things pleasant across as many dev envs as we can.
* Beyond that, there will likely be a longer tail of smaller issues or missing things even after we make the switch.
* I'll put some effort into the side car experiment for another month or two perhaps (maybe not much in November)
* If things are not looking like we will be able to flip the switch, I'll look at pulling out Gradle.

- Mark


--


--


--


--


--
--


--
Reply | Threaded
Open this post in threaded view
|

Re: The Lucene Solr Gradle Build Game plan

Gus Heck
Haven't looked at your code, but it sounds a bit scary for you (or whomever comes after you) to touch my ~/.gradle/gradle.properties. Sometimes I have some pretty important stuff there.... Can we use the project specific gradle.properties in the project root dir and make it .gitignore?

On Sun, Sep 15, 2019 at 8:47 PM Mark Miller <[hidden email]> wrote:
Okay, there is now a task called defaultUserConfig

You can do gw defaultUserConfig to set some recommended settings in ~/.gradle/gradle.properties. By default they should be more relaxed, but we can tune as needed.

For better build performance but also more resource usage you can do:

gw defaultUserConfig --style=aggressive

Mark

On Sun, Sep 15, 2019 at 5:50 PM Mark Miller <[hidden email]> wrote:
By the way, I did hear about the hack day and some Gradle testing, which is great, and very useful. Totally needed, but we also need a bit of a more deep core developer view of things vs the old build as well. The type of stuff that’s much harder to tease out than verifying all the new build targets and such. Of course a lot of that can come after we switch, but I have a sneaky feeling some core devs will have deep opinions about certain things. 

Ill add a new task for default config setup. 

Mark

On Sun, Sep 15, 2019 at 5:12 PM Mark Miller <[hidden email]> wrote:
I'll just detail it out here as well:

You can configure parallelism in ~/.gradle/gradle.properties

org.gradle.workers.max=2
tests_jvms=5

org.gradle.workers.max is controlled by gradle and defaults to the number of cores detected - I wish I could change to divided by 2.
test_jvms is controlled by us and defaults to the number of cores detected / 2.

org.gradle.workers.max  controls the total number of jvms that will be run in parallel - for tasks or tests or whatever gradle is doing.
test_jvms controls how many parallel jvms a module will use for tests, but that is also limited by org.gradle.workers.max

You should try setting both to cores / 2 and work down from there if needed.

When running tests across multiple modules, org.gradle.workers.max is the actual limit for test jvms spun up because each module is only limited to test_jvms, but ALL tasks are limited to org.gradle.workers.max and module tasks are run in parallel by default.

By setting them the same, we get similar behavior whether we run tests from the root directory of the project (all tests) or from a single module (say solr-core).

On Sun, Sep 15, 2019 at 5:03 PM Mark Miller <[hidden email]> wrote:
I've added more about that here: https://cwiki.apache.org/confluence/display/SOLR/Intro+to+the+Gradle+build

It's configurable, but difficult for us to choose a default based on cores as far as I can tell and gradles default, which is based on cores detected, is too high, especially because of hyperthreading.

Probably the best we can do is default it to a hard 2 or 4 and let people raise it depending on what wait you want to error. In both cases the majority of people will want to change it.

On Sun, Sep 15, 2019 at 3:39 PM Gus Heck <[hidden email]> wrote:
Not sure if you heard, but about a half a dozen folks tried it out on macs and one on windows at the hack day on Tuesday before Activate. It caused some scrambling for sharing of power bricks (a single run of the tests eats 70% of a fully charged 2018 macbook pro battery in 45 min), but the good news is it only failed on well known flaky tests and on the one windows machine and that in the PDF building for the ref guide (and there was that small bit with the error message and the AtomicBoolean that I fixed). I've heard some opinions that maybe we don't need the PDF version in the future, The one bit of feedback that came out of it was it would be nice to have a task that tweaked the configs things to not peg the processor quite so efficiently :). Certainly we do want the mode for full utilization to be available, but a background mode would be good too in case folks need to do other work.

-Gus

On Sun, Sep 15, 2019 at 3:24 PM Mark Miller <[hidden email]> wrote:
Okay, I've tried to spread that link a little on social media as well.

Please do some experimentation. Especially those of you that use or know more esoteric things about the build. The basics are pretty solid, most have been working for months now, it's the corners and crannies that I'm more concerned about. You could have built and run tests a few months ago and said, nice, it's working, but then I've done 10,000 things since then that were necessary. You may easily not realize things are a problem until you really dig into something.

The more that devs can start trying out Gradle now, the less overlap of the two build systems we will need.

The idea of the overlap is that it will almost force many of us to start trying things out - at which point we will start to understand any key things that are missing, key problems or bugs, etc. If I just make the switch and let the cards fall were they may, those cards are almost certainly going to be very disruptive and annoying to a lot of people.

It will also give us a chance to start rolling CI and other tools over to using the Gradle build while ant+ivy are still available and in charge. I don't think it's a great idea to try and do this all at one moment, that will be difficult to coordinate and be more disruptive to less interested devs. We would like to keep the dual build situation contained and short though. That's why I have a plan to pull out if we don't make enough progress by a month or twos time (closer to a month). 

If we have enough experimentation ahead of time, the overlap can be very short - we start moving things like jenkins jobs over and when we are done and confident the world is not ending, we make some adjustments so that gradle owns the build (eg the dependency tree) and then remove ant+ivy+maven.

Some things like the smoke tester and what not *can* come shortly after we switch I think - we have until the 9 release to truly get everything in order (keeping in mind that we are still developing so some things are still critical). But we need CI and other basics all moved over when we flip the ownership switch.

- Mark

On Sun, Sep 15, 2019 at 12:28 PM Mark Miller <[hidden email]> wrote:
I've started to put together a little guide to help people ramp up here: https://cwiki.apache.org/confluence/display/SOLR/Intro+to+the+Gradle+build

On Sat, Sep 14, 2019 at 9:09 PM Mark Miller <[hidden email]> wrote:
https://issues.apache.org/jira/browse/SOLR-13452
Update the lucene-solr build from Ivy+Ant+Maven (shadow build) to Gradle.

Hey all.

Here is my Lucene and Solr 'move to Gradle' plans.

* I plan on trying to commit the Gradle build by 9/30
* In the meantime I'll do a bit more work on packaging, publishing, and dependencies.
* It would be great if others could start digging in a bit as well.
* I plan to try and sidecar Gradle to begin with. Once that happens, I really need others to start digging in as we have a lot to do (switch ci and other tools to the new build, figure out new release docs and issues, finish the Solr documentation module (I'll try and get to that earlier than later), and generally make things pleasant across as many dev envs as we can.
* Beyond that, there will likely be a longer tail of smaller issues or missing things even after we make the switch.
* I'll put some effort into the side car experiment for another month or two perhaps (maybe not much in November)
* If things are not looking like we will be able to flip the switch, I'll look at pulling out Gradle.

- Mark


--


--


--


--


--
--


--


--
Reply | Threaded
Open this post in threaded view
|

Re: The Lucene Solr Gradle Build Game plan

Mark Miller-3
That's against Gradle best practices, if you look it up, this is the way Gradle is intended to work. Settings like 

That task description can warn about editing your file, but I feel it's pretty safe given that you have to decide to use it.

We don't remove anything and we don't change controversial settings (unless they break us, like configure on demand).

Settings like org.gradle.workers.max are used by the Daemon and meant to be pretty universal as Daemons are intended to be reused.

Another property is ours - if you have overridden it, asking for these settings changes will change it. Good?

Configure on demand will break us and is experimental and is noted around a variety of bugs, so we disable it.

Sure, just smashing up someones ~/.gradle/gradle.properties is not nice, but making required and performance adjustments at request seems great to me.

- Mark

On Sun, Sep 15, 2019 at 10:56 PM Gus Heck <[hidden email]> wrote:
Haven't looked at your code, but it sounds a bit scary for you (or whomever comes after you) to touch my ~/.gradle/gradle.properties. Sometimes I have some pretty important stuff there.... Can we use the project specific gradle.properties in the project root dir and make it .gitignore?

On Sun, Sep 15, 2019 at 8:47 PM Mark Miller <[hidden email]> wrote:
Okay, there is now a task called defaultUserConfig

You can do gw defaultUserConfig to set some recommended settings in ~/.gradle/gradle.properties. By default they should be more relaxed, but we can tune as needed.

For better build performance but also more resource usage you can do:

gw defaultUserConfig --style=aggressive

Mark

On Sun, Sep 15, 2019 at 5:50 PM Mark Miller <[hidden email]> wrote:
By the way, I did hear about the hack day and some Gradle testing, which is great, and very useful. Totally needed, but we also need a bit of a more deep core developer view of things vs the old build as well. The type of stuff that’s much harder to tease out than verifying all the new build targets and such. Of course a lot of that can come after we switch, but I have a sneaky feeling some core devs will have deep opinions about certain things. 

Ill add a new task for default config setup. 

Mark

On Sun, Sep 15, 2019 at 5:12 PM Mark Miller <[hidden email]> wrote:
I'll just detail it out here as well:

You can configure parallelism in ~/.gradle/gradle.properties

org.gradle.workers.max=2
tests_jvms=5

org.gradle.workers.max is controlled by gradle and defaults to the number of cores detected - I wish I could change to divided by 2.
test_jvms is controlled by us and defaults to the number of cores detected / 2.

org.gradle.workers.max  controls the total number of jvms that will be run in parallel - for tasks or tests or whatever gradle is doing.
test_jvms controls how many parallel jvms a module will use for tests, but that is also limited by org.gradle.workers.max

You should try setting both to cores / 2 and work down from there if needed.

When running tests across multiple modules, org.gradle.workers.max is the actual limit for test jvms spun up because each module is only limited to test_jvms, but ALL tasks are limited to org.gradle.workers.max and module tasks are run in parallel by default.

By setting them the same, we get similar behavior whether we run tests from the root directory of the project (all tests) or from a single module (say solr-core).

On Sun, Sep 15, 2019 at 5:03 PM Mark Miller <[hidden email]> wrote:
I've added more about that here: https://cwiki.apache.org/confluence/display/SOLR/Intro+to+the+Gradle+build

It's configurable, but difficult for us to choose a default based on cores as far as I can tell and gradles default, which is based on cores detected, is too high, especially because of hyperthreading.

Probably the best we can do is default it to a hard 2 or 4 and let people raise it depending on what wait you want to error. In both cases the majority of people will want to change it.

On Sun, Sep 15, 2019 at 3:39 PM Gus Heck <[hidden email]> wrote:
Not sure if you heard, but about a half a dozen folks tried it out on macs and one on windows at the hack day on Tuesday before Activate. It caused some scrambling for sharing of power bricks (a single run of the tests eats 70% of a fully charged 2018 macbook pro battery in 45 min), but the good news is it only failed on well known flaky tests and on the one windows machine and that in the PDF building for the ref guide (and there was that small bit with the error message and the AtomicBoolean that I fixed). I've heard some opinions that maybe we don't need the PDF version in the future, The one bit of feedback that came out of it was it would be nice to have a task that tweaked the configs things to not peg the processor quite so efficiently :). Certainly we do want the mode for full utilization to be available, but a background mode would be good too in case folks need to do other work.

-Gus

On Sun, Sep 15, 2019 at 3:24 PM Mark Miller <[hidden email]> wrote:
Okay, I've tried to spread that link a little on social media as well.

Please do some experimentation. Especially those of you that use or know more esoteric things about the build. The basics are pretty solid, most have been working for months now, it's the corners and crannies that I'm more concerned about. You could have built and run tests a few months ago and said, nice, it's working, but then I've done 10,000 things since then that were necessary. You may easily not realize things are a problem until you really dig into something.

The more that devs can start trying out Gradle now, the less overlap of the two build systems we will need.

The idea of the overlap is that it will almost force many of us to start trying things out - at which point we will start to understand any key things that are missing, key problems or bugs, etc. If I just make the switch and let the cards fall were they may, those cards are almost certainly going to be very disruptive and annoying to a lot of people.

It will also give us a chance to start rolling CI and other tools over to using the Gradle build while ant+ivy are still available and in charge. I don't think it's a great idea to try and do this all at one moment, that will be difficult to coordinate and be more disruptive to less interested devs. We would like to keep the dual build situation contained and short though. That's why I have a plan to pull out if we don't make enough progress by a month or twos time (closer to a month). 

If we have enough experimentation ahead of time, the overlap can be very short - we start moving things like jenkins jobs over and when we are done and confident the world is not ending, we make some adjustments so that gradle owns the build (eg the dependency tree) and then remove ant+ivy+maven.

Some things like the smoke tester and what not *can* come shortly after we switch I think - we have until the 9 release to truly get everything in order (keeping in mind that we are still developing so some things are still critical). But we need CI and other basics all moved over when we flip the ownership switch.

- Mark

On Sun, Sep 15, 2019 at 12:28 PM Mark Miller <[hidden email]> wrote:
I've started to put together a little guide to help people ramp up here: https://cwiki.apache.org/confluence/display/SOLR/Intro+to+the+Gradle+build

On Sat, Sep 14, 2019 at 9:09 PM Mark Miller <[hidden email]> wrote:
https://issues.apache.org/jira/browse/SOLR-13452
Update the lucene-solr build from Ivy+Ant+Maven (shadow build) to Gradle.

Hey all.

Here is my Lucene and Solr 'move to Gradle' plans.

* I plan on trying to commit the Gradle build by 9/30
* In the meantime I'll do a bit more work on packaging, publishing, and dependencies.
* It would be great if others could start digging in a bit as well.
* I plan to try and sidecar Gradle to begin with. Once that happens, I really need others to start digging in as we have a lot to do (switch ci and other tools to the new build, figure out new release docs and issues, finish the Solr documentation module (I'll try and get to that earlier than later), and generally make things pleasant across as many dev envs as we can.
* Beyond that, there will likely be a longer tail of smaller issues or missing things even after we make the switch.
* I'll put some effort into the side car experiment for another month or two perhaps (maybe not much in November)
* If things are not looking like we will be able to flip the switch, I'll look at pulling out Gradle.

- Mark


--


--


--


--


--
--


--


--


--
Reply | Threaded
Open this post in threaded view
|

Re: The Lucene Solr Gradle Build Game plan

Mark Miller-3
Maybe making a backup first would improve things?

My first feeling was just let the user do this, but I work in a bunch of envs and it's very nice to have.

At worst the task description and any doc can say WARNING!: we are going to edit your file if you do this, there will be a .bak or something.

Mark

On Mon, Sep 16, 2019 at 1:36 PM Mark Miller <[hidden email]> wrote:
That's against Gradle best practices, if you look it up, this is the way Gradle is intended to work. Settings like 

That task description can warn about editing your file, but I feel it's pretty safe given that you have to decide to use it.

We don't remove anything and we don't change controversial settings (unless they break us, like configure on demand).

Settings like org.gradle.workers.max are used by the Daemon and meant to be pretty universal as Daemons are intended to be reused.

Another property is ours - if you have overridden it, asking for these settings changes will change it. Good?

Configure on demand will break us and is experimental and is noted around a variety of bugs, so we disable it.

Sure, just smashing up someones ~/.gradle/gradle.properties is not nice, but making required and performance adjustments at request seems great to me.

- Mark

On Sun, Sep 15, 2019 at 10:56 PM Gus Heck <[hidden email]> wrote:
Haven't looked at your code, but it sounds a bit scary for you (or whomever comes after you) to touch my ~/.gradle/gradle.properties. Sometimes I have some pretty important stuff there.... Can we use the project specific gradle.properties in the project root dir and make it .gitignore?

On Sun, Sep 15, 2019 at 8:47 PM Mark Miller <[hidden email]> wrote:
Okay, there is now a task called defaultUserConfig

You can do gw defaultUserConfig to set some recommended settings in ~/.gradle/gradle.properties. By default they should be more relaxed, but we can tune as needed.

For better build performance but also more resource usage you can do:

gw defaultUserConfig --style=aggressive

Mark

On Sun, Sep 15, 2019 at 5:50 PM Mark Miller <[hidden email]> wrote:
By the way, I did hear about the hack day and some Gradle testing, which is great, and very useful. Totally needed, but we also need a bit of a more deep core developer view of things vs the old build as well. The type of stuff that’s much harder to tease out than verifying all the new build targets and such. Of course a lot of that can come after we switch, but I have a sneaky feeling some core devs will have deep opinions about certain things. 

Ill add a new task for default config setup. 

Mark

On Sun, Sep 15, 2019 at 5:12 PM Mark Miller <[hidden email]> wrote:
I'll just detail it out here as well:

You can configure parallelism in ~/.gradle/gradle.properties

org.gradle.workers.max=2
tests_jvms=5

org.gradle.workers.max is controlled by gradle and defaults to the number of cores detected - I wish I could change to divided by 2.
test_jvms is controlled by us and defaults to the number of cores detected / 2.

org.gradle.workers.max  controls the total number of jvms that will be run in parallel - for tasks or tests or whatever gradle is doing.
test_jvms controls how many parallel jvms a module will use for tests, but that is also limited by org.gradle.workers.max

You should try setting both to cores / 2 and work down from there if needed.

When running tests across multiple modules, org.gradle.workers.max is the actual limit for test jvms spun up because each module is only limited to test_jvms, but ALL tasks are limited to org.gradle.workers.max and module tasks are run in parallel by default.

By setting them the same, we get similar behavior whether we run tests from the root directory of the project (all tests) or from a single module (say solr-core).

On Sun, Sep 15, 2019 at 5:03 PM Mark Miller <[hidden email]> wrote:
I've added more about that here: https://cwiki.apache.org/confluence/display/SOLR/Intro+to+the+Gradle+build

It's configurable, but difficult for us to choose a default based on cores as far as I can tell and gradles default, which is based on cores detected, is too high, especially because of hyperthreading.

Probably the best we can do is default it to a hard 2 or 4 and let people raise it depending on what wait you want to error. In both cases the majority of people will want to change it.

On Sun, Sep 15, 2019 at 3:39 PM Gus Heck <[hidden email]> wrote:
Not sure if you heard, but about a half a dozen folks tried it out on macs and one on windows at the hack day on Tuesday before Activate. It caused some scrambling for sharing of power bricks (a single run of the tests eats 70% of a fully charged 2018 macbook pro battery in 45 min), but the good news is it only failed on well known flaky tests and on the one windows machine and that in the PDF building for the ref guide (and there was that small bit with the error message and the AtomicBoolean that I fixed). I've heard some opinions that maybe we don't need the PDF version in the future, The one bit of feedback that came out of it was it would be nice to have a task that tweaked the configs things to not peg the processor quite so efficiently :). Certainly we do want the mode for full utilization to be available, but a background mode would be good too in case folks need to do other work.

-Gus

On Sun, Sep 15, 2019 at 3:24 PM Mark Miller <[hidden email]> wrote:
Okay, I've tried to spread that link a little on social media as well.

Please do some experimentation. Especially those of you that use or know more esoteric things about the build. The basics are pretty solid, most have been working for months now, it's the corners and crannies that I'm more concerned about. You could have built and run tests a few months ago and said, nice, it's working, but then I've done 10,000 things since then that were necessary. You may easily not realize things are a problem until you really dig into something.

The more that devs can start trying out Gradle now, the less overlap of the two build systems we will need.

The idea of the overlap is that it will almost force many of us to start trying things out - at which point we will start to understand any key things that are missing, key problems or bugs, etc. If I just make the switch and let the cards fall were they may, those cards are almost certainly going to be very disruptive and annoying to a lot of people.

It will also give us a chance to start rolling CI and other tools over to using the Gradle build while ant+ivy are still available and in charge. I don't think it's a great idea to try and do this all at one moment, that will be difficult to coordinate and be more disruptive to less interested devs. We would like to keep the dual build situation contained and short though. That's why I have a plan to pull out if we don't make enough progress by a month or twos time (closer to a month). 

If we have enough experimentation ahead of time, the overlap can be very short - we start moving things like jenkins jobs over and when we are done and confident the world is not ending, we make some adjustments so that gradle owns the build (eg the dependency tree) and then remove ant+ivy+maven.

Some things like the smoke tester and what not *can* come shortly after we switch I think - we have until the 9 release to truly get everything in order (keeping in mind that we are still developing so some things are still critical). But we need CI and other basics all moved over when we flip the ownership switch.

- Mark

On Sun, Sep 15, 2019 at 12:28 PM Mark Miller <[hidden email]> wrote:
I've started to put together a little guide to help people ramp up here: https://cwiki.apache.org/confluence/display/SOLR/Intro+to+the+Gradle+build

On Sat, Sep 14, 2019 at 9:09 PM Mark Miller <[hidden email]> wrote:
https://issues.apache.org/jira/browse/SOLR-13452
Update the lucene-solr build from Ivy+Ant+Maven (shadow build) to Gradle.

Hey all.

Here is my Lucene and Solr 'move to Gradle' plans.

* I plan on trying to commit the Gradle build by 9/30
* In the meantime I'll do a bit more work on packaging, publishing, and dependencies.
* It would be great if others could start digging in a bit as well.
* I plan to try and sidecar Gradle to begin with. Once that happens, I really need others to start digging in as we have a lot to do (switch ci and other tools to the new build, figure out new release docs and issues, finish the Solr documentation module (I'll try and get to that earlier than later), and generally make things pleasant across as many dev envs as we can.
* Beyond that, there will likely be a longer tail of smaller issues or missing things even after we make the switch.
* I'll put some effort into the side car experiment for another month or two perhaps (maybe not much in November)
* If things are not looking like we will be able to flip the switch, I'll look at pulling out Gradle.

- Mark


--


--


--


--


--
--


--


--


--


--
Reply | Threaded
Open this post in threaded view
|

Re: The Lucene Solr Gradle Build Game plan

Chris Hostetter-3
In reply to this post by Mark Miller-3

Some misc questions after playing around with gradle on this branch for a
bit today in no particular order...

: tests_jvms=5
        ...
: test_jvms is controlled by us and defaults to the number of cores detected
: / 2.

If tests_jvms is a lucene/solr specific property, that needs to
be in a users "multi-project" "~/.gradle" file ... shouldn't we name it
namespace it with something like "lucene_solr_test_jvms" to make that
clear reduce future confusion/collsion?  (as i anticipate this wo't be the
last prop we may need)

How do we get "reproduce with" type output (by default) when a test fails?

For that matter, how do we get *all* of the logging from failed tests to
be written to stdout/stderr when a test fail?  ... this is pretty
important for making jenkin's console log a good "one stop shop" for
understanding everything that went wrong in a build.  

("--debug" and "--info" seem to do this, but they write a *TON* more
gradle specific shit then "ant test" type logging use to produce by
default, and don't care if the test passes or not ... which would make
them way too excessively verbose for a jenkins build)

What's the plan as far as things like "ant beast", "-Dtests.dups",
"-Dtests.iters" & "-Dtests.jvms" ? ... those types of options are all
pretty critical for diagnosing and troubleshooting failures.  
("-Dtests.jvms=1" is important for figuring out if/how the execution of
one test might polute static variables in the JVM that cause a failure in
a subsequent class in the sane JVM)

is there a simple option to prevent gradle from using curses even though
it detects it's being run in a tty?


-Hoss
http://www.lucidworks.com/

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: The Lucene Solr Gradle Build Game plan

Mark Miller-3

This is no reproduce output yet, in my mind that is waiting with the tests seed work that I started on but needs review and finishing.

tests_jvms is jvms and no user param naming is yet thought out or final when it comes to modifying the build. I've instead stuck to exposing few things and expecting them to get exposed as needed.

tests_jvms is basically a default translation from ant, I'm hoping someone else want's to own what happens here.

bq. how do we get *all* of the logging from failed tests to be written to stdout/stderr when a test fail?

Most things are pretty easy in Gradle. If you want to do it or want me to do it, please file a JIRA. If it's not a quick fix for someone, it would be great to start tracking what each individual thinks is critical in JIRA.

All the test stuff needs review and some tweaks (has not changed sine last email thread which you may not have caught), last I remember Dawid signed up for it starting around today (15th) ;) See previous email chain.

bq. What's the plan as far as things like "ant beast"

beast stuff will come later

tests.jvms is tests_jvms :) The other settings along these lines are either part of the test Runner (not most) or the Ant test launcher from RandomizedTesting. We will have some form of beasting, but that's almost last in line in terms of what I'm doing now.

- Mark

On Mon, Sep 16, 2019 at 1:59 PM Chris Hostetter <[hidden email]> wrote:

Some misc questions after playing around with gradle on this branch for a
bit today in no particular order...

: tests_jvms=5
        ...
: test_jvms is controlled by us and defaults to the number of cores detected
: / 2.

If tests_jvms is a lucene/solr specific property, that needs to
be in a users "multi-project" "~/.gradle" file ... shouldn't we name it
namespace it with something like "lucene_solr_test_jvms" to make that
clear reduce future confusion/collsion?  (as i anticipate this wo't be the
last prop we may need)

How do we get "reproduce with" type output (by default) when a test fails?

For that matter, how do we get *all* of the logging from failed tests to
be written to stdout/stderr when a test fail?  ... this is pretty
important for making jenkin's console log a good "one stop shop" for
understanding everything that went wrong in a build. 

("--debug" and "--info" seem to do this, but they write a *TON* more
gradle specific shit then "ant test" type logging use to produce by
default, and don't care if the test passes or not ... which would make
them way too excessively verbose for a jenkins build)

What's the plan as far as things like "ant beast", "-Dtests.dups",
"-Dtests.iters" & "-Dtests.jvms" ? ... those types of options are all
pretty critical for diagnosing and troubleshooting failures. 
("-Dtests.jvms=1" is important for figuring out if/how the execution of
one test might polute static variables in the JVM that cause a failure in
a subsequent class in the sane JVM)

is there a simple option to prevent gradle from using curses even though
it detects it's being run in a tty?


-Hoss
http://www.lucidworks.com/

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]



--
Reply | Threaded
Open this post in threaded view
|

Re: The Lucene Solr Gradle Build Game plan

Dawid Weiss-2
> [...] last I remember Dawid signed up for it starting around today (15th) ;)

Ah... so you remembered?... Kind of hoped you forgot. :)

I'll take a look and try to tackle some of Chris's questions. Are we
free to commit to that gradle branch or do you prefer PRs?

D.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: The Lucene Solr Gradle Build Game plan

Cassandra Targett
I got started on the Ref Guide build last night. The biggest change there is to use the asciidoctor-gradle-plugin instead of using the asciidoctor-ant plugin.

So far I’ve got it working enough to build a single page of the Ref Guide into a PDF file. Baby steps ;)

By itself that only gets us the PDF, while our HTML is built with several Ruby gems which we had to require people to install locally. However, the asciidoctor-gradle plugin includes JRuby, so fingers crossed we’ll finally be able to get that working and end up in a better place than we are today, and with a more unified build configuration than we currently have.

I’m not ready to push anything to the branch yet, but hopefully will be in the next day or two.

Cassandra
On Sep 17, 2019, 5:02 AM -0500, Dawid Weiss <[hidden email]>, wrote:
[...] last I remember Dawid signed up for it starting around today (15th) ;)

Ah... so you remembered?... Kind of hoped you forgot. :)

I'll take a look and try to tackle some of Chris's questions. Are we
free to commit to that gradle branch or do you prefer PRs?

D.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: The Lucene Solr Gradle Build Game plan

Mark Miller-3
bq. Are we free to commit to that gradle branch or do you prefer PRs?

Commit away unless you are looking for feedback and a PR makes it easier.

I held onto it for long enough :) Time to get some more blood in this thing.

Also, if there are any things or parts that are important and you don't have time for, file a JIRA and assign me and I will do it. Largely, I am looking more for input and collaboration and what's more critically missing then manual labor. At the same time, we need some labor too, because if this goes in with me doing 99% of it, there is something that smells wrong about moving.

bq. I’m not ready to push anything to the branch yet, but hopefully will be in the next day or two.

Thanks so much Cassandra! Let me know if I can help on any trouble.

- Mark

On Tue, Sep 17, 2019 at 12:31 PM Cassandra Targett <[hidden email]> wrote:
I got started on the Ref Guide build last night. The biggest change there is to use the asciidoctor-gradle-plugin instead of using the asciidoctor-ant plugin.

So far I’ve got it working enough to build a single page of the Ref Guide into a PDF file. Baby steps ;)

By itself that only gets us the PDF, while our HTML is built with several Ruby gems which we had to require people to install locally. However, the asciidoctor-gradle plugin includes JRuby, so fingers crossed we’ll finally be able to get that working and end up in a better place than we are today, and with a more unified build configuration than we currently have.

I’m not ready to push anything to the branch yet, but hopefully will be in the next day or two.

Cassandra
On Sep 17, 2019, 5:02 AM -0500, Dawid Weiss <[hidden email]>, wrote:
[...] last I remember Dawid signed up for it starting around today (15th) ;)

Ah... so you remembered?... Kind of hoped you forgot. :)

I'll take a look and try to tackle some of Chris's questions. Are we
free to commit to that gradle branch or do you prefer PRs?

D.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]



--
Reply | Threaded
Open this post in threaded view
|

Re: The Lucene Solr Gradle Build Game plan

Gus Heck
In reply to this post by Cassandra Targett
I mentioned when we chatted at activate that I had a build in the past that was doing some pulling of gems... Looked back and it was actually using https://github.com/robfletcher/gradle-compass which was doing the gem management. Seems to use https://github.com/jruby-gradle/jruby-gradle-plugin to pull gems (if I'm reading their code right). That could be helpful in automating the requirements for the html ref guide.

On Tue, Sep 17, 2019 at 1:32 PM Cassandra Targett <[hidden email]> wrote:
I got started on the Ref Guide build last night. The biggest change there is to use the asciidoctor-gradle-plugin instead of using the asciidoctor-ant plugin.

So far I’ve got it working enough to build a single page of the Ref Guide into a PDF file. Baby steps ;)

By itself that only gets us the PDF, while our HTML is built with several Ruby gems which we had to require people to install locally. However, the asciidoctor-gradle plugin includes JRuby, so fingers crossed we’ll finally be able to get that working and end up in a better place than we are today, and with a more unified build configuration than we currently have.

I’m not ready to push anything to the branch yet, but hopefully will be in the next day or two.

Cassandra
On Sep 17, 2019, 5:02 AM -0500, Dawid Weiss <[hidden email]>, wrote:
[...] last I remember Dawid signed up for it starting around today (15th) ;)

Ah... so you remembered?... Kind of hoped you forgot. :)

I'll take a look and try to tackle some of Chris's questions. Are we
free to commit to that gradle branch or do you prefer PRs?

D.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]



--
Reply | Threaded
Open this post in threaded view
|

Re: The Lucene Solr Gradle Build Game plan

Dawid Weiss-2
In reply to this post by Chris Hostetter-3
I've been eyeballing Mark's impressive work on the build system in
Gradle. Hoss asked a
numer of questions that relate to tests and I feel I should address
them, even if some poorly. Here it comes, inlined with Hoss's
questions.

> If tests_jvms is a lucene/solr specific property, that needs to be in a users "multi-project" "~/.gradle" file

Yeah... I'm not at all a fan of modifying the global gradle settings
file. While it may be ok for settings that only apply to Lucene (if
prefixed) there are other things like enabling or disabling the daemon
which should be a matter of project-local settings. Ideally, I'd like
to see something like:

checkout/gradle-defaults.properties [versioned]
checkout/gradle-local.properties [developer-tweaks, not versioned]

but short of declaring a custom "user home" for gradle (-g switch) I
don't see how this can be achieved. Perhaps the initial checkout could
not carry gradle.properties at all and the build would require an
initial setup via Mark's "defaultUserConfig". This could then write
(unversioned!) gradle.properties with the selected defaults within the
checkout folder (so there'd be no need to tweak the global file).
People can then tweak it to their liking or call defaultUserConfig
with appropriate options again to overwrite the changes.

> How do we get "reproduce with" type output (by default) when a test fails?

This should be doable -- we do it from Ant after all. You can pass
properties to gradle via "-P" so this can be used in the same way we
pass them to ant. In fact, it already works:

gradlew :lucene:lucene-core:test --tests TestDemo -Ptests.seed=deadbeef

> For that matter, how do we get *all* of the logging from failed tests to be written to stdout/stderr when a test fail?

There probably is an API in gradle to do this. I know Ryan wrote a
runner for Elastic a long time ago so there may be even code to look
at. I agree it is quite important to get this to work, especially for
CI logs.

> What's the plan as far as things like "ant beast", "-Dtests.dups", "-Dtests.iters" & "-Dtests.jvms" ?

Reiteration (tests.iters) is part of the test runner so it'd already
work if that property was passed to it (via -P). It isn't at the
moment -- I'll work on it.

As for tests.dups and tests.jvms this will have to be emulated somehow
at Gradle level. Don't know how though. I'm sure there'll be ways --
this is essentially a programming language.

> ("-Dtests.jvms=1" is important for figuring out if/how the execution of one test might polute static variables in the JVM

I don't think it'll work even if you have one JVM unless Gradle test
runner's order is deterministic... which I'm not sure it is, even with
a single JVM that runs the tests.

> is there a simple option to prevent gradle from using curses even though it detects it's being run in a tty?

You can pass --console=plain to gradle, use -Dorg.gradle.console=plain
or specify this property in any other way (via environment variable,
etc.). See here:

https://docs.gradle.org/current/userguide/build_environment.html

Dawid




On Mon, Sep 16, 2019 at 8:59 PM Chris Hostetter
<[hidden email]> wrote:

>
>
> Some misc questions after playing around with gradle on this branch for a
> bit today in no particular order...
>
> : tests_jvms=5
>         ...
> : test_jvms is controlled by us and defaults to the number of cores detected
> : / 2.
>
> If tests_jvms is a lucene/solr specific property, that needs to
> be in a users "multi-project" "~/.gradle" file ... shouldn't we name it
> namespace it with something like "lucene_solr_test_jvms" to make that
> clear reduce future confusion/collsion?  (as i anticipate this wo't be the
> last prop we may need)
>
> How do we get "reproduce with" type output (by default) when a test fails?
>
> For that matter, how do we get *all* of the logging from failed tests to
> be written to stdout/stderr when a test fail?  ... this is pretty
> important for making jenkin's console log a good "one stop shop" for
> understanding everything that went wrong in a build.
>
> ("--debug" and "--info" seem to do this, but they write a *TON* more
> gradle specific shit then "ant test" type logging use to produce by
> default, and don't care if the test passes or not ... which would make
> them way too excessively verbose for a jenkins build)
>
> What's the plan as far as things like "ant beast", "-Dtests.dups",
> "-Dtests.iters" & "-Dtests.jvms" ? ... those types of options are all
> pretty critical for diagnosing and troubleshooting failures.
> ("-Dtests.jvms=1" is important for figuring out if/how the execution of
> one test might polute static variables in the JVM that cause a failure in
> a subsequent class in the sane JVM)
>
> is there a simple option to prevent gradle from using curses even though
> it detects it's being run in a tty?
>
>
> -Hoss
> http://www.lucidworks.com/
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: The Lucene Solr Gradle Build Game plan

Chris Hostetter-3

: which should be a matter of project-local settings. Ideally, I'd like
: to see something like:
:
: checkout/gradle-defaults.properties [versioned]
: checkout/gradle-local.properties [developer-tweaks, not versioned]
:
: but short of declaring a custom "user home" for gradle (-g switch) I
: don't see how this can be achieved. Perhaps the initial checkout could

I gather from your comments (and from the link you mentioned in another
question) that gradle does not have a direct equivilent of ant's...

  <property file="${user.home}/lucene.build.properties"/>

...but some naive google searching suggests it's possible to do something
similar using "apply from" to load in a *.gradle file using an arbitrary
path, which could contain property declarations which would then evidently
by overlayed on the existing groovy file?
(maybe?)

Ex: this in our build.gradle file...

        apply from: '$USER_HOME/lucene.properties.gradle'

...and this in my ~/lucene.properties.gradle file...

        ext {
            test_jvms = 42
        }

...would that work?


: > ("-Dtests.jvms=1" is important for figuring out if/how the execution
: of one test might polute static variables in the JVM
:
: I don't think it'll work even if you have one JVM unless Gradle test
: runner's order is deterministic... which I'm not sure it is, even with
: a single JVM that runs the tests.

FWIW: I'm not sure if the deterministic ordering is that important --
although you reminded me that being able to *review* the list of tests
that ran in the same JVM when a failure happens is helpful...

(i don't know that i've ever tried to rerun that exact list of every test
that ran in the same JVM in the same order as what was reported -- or even
how i would if i wanted to other then with a really fucking complicated
-Dtests.class=... param)

I'd guess that 90% of the time when i currently use "-Dtests.jvms=1" it's
when running a single test class with -Dtests.dups=N to confirm & fix
shitty use of statics in BeforeClass/AfterClass methods (typically
problematic if/when the jenkins [repro] target does something like
-Dtests.dups=5 -DtestcaseSomeTestThatFailed but defaults to using
tests.jvms=3, so we see 2/5 suite level failures that look nothing like
whatever the original failure was that jenkins was attempting to
reproduce)

the 10% of the time -- where that's not the issue, and a failure in one
test really was caused byt some previous test in the same JVM mucking up
some static variable state -- eyeballing the list of tests that also ran
in that JVM usually helps narrow down the list of suspects, and then i
just try beasting those 2 (or 3) tests together with "-Dtests.jvms=1", not
worrying about the order and just trusting that if i'm right eventually
they'll be run in the problematic order.


-Hoss
http://www.lucidworks.com/

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: The Lucene Solr Gradle Build Game plan

Mark Miller-3

On Wed, Sep 18, 2019 at 10:46 AM Dawid Weiss <[hidden email]> wrote:

> If tests_jvms is a lucene/solr specific property, that needs to be in a users "multi-project" "~/.gradle" file

Yeah... I'm not at all a fan of modifying the global gradle settings
file. While it may be ok for settings that only apply to Lucene (if
prefixed) there are other things like enabling or disabling the daemon
which should be a matter of project-local settings. Ideally, I'd like
to see something like:

I find the Gradle way of doing things odd, but I knew better than to try and go directly against them.

So none of this is really my choosing, I went with how you are supposed to do it.

When it comes to allowing another prop file just for our project, I'm all for it - just not the standard gradle.properties that is supposed to be committed in a gradle project.

You cannot override any settings from your gradle home file regardless, so it's a little non intuitive - gradle home props rule all.

Anyway, I have exposed like one of our properties, so no, it's not thought out or how it's going to be at all. Probably that property should be name spaced beyond tests_jvms (translated from tests.jvms), but then we might want to support a short cut notation for command line.

Sucking in a user prop file like we do with ant sounds great, just have to get used to the fact that you cannot override your global settings. Trying to subvert that in some way def feels like a bad path.

Anyway, I've never found it to be a problem. I didn't expect everyone would customize our project settings in their gradle home file, I usually put stuff there that I want to apply to all my checkouts (tests.jvms is actually one of them), but all that is very individual.

You can also set these on the command line - it's mostly the same as ant. If you leave stuff out of your home gradle file and just add to some file we suck in, ezpeasy. I haven't prescribed or enforced anything here.

There will be more/better getting started doc and more finished stuff the closer we get.

> What's the plan as far as things like "ant beast", "-Dtests.dups", "-Dtests.iters" & "-Dtests.jvms" ?

I've been having plenty of issues with just regular tests. Gradle statically assigns all tests in basically fs order to N JVMs. If you run tests against all the Solr modules, it's pretty sweet because it doesn't just use a few executors on the small modules and go a module at a time - it does them together more and less and keeps things busy.
But running tests within a module, or even running all modules tests, was no good. You have solr-core tests coming late, you have many tests that pop up from short runs to minutes+ (often tests with many sub class tests), and you have a static JVM assignment for each test. So basically, it's easy to be running long tests at the end of a run and use few JVM's, and with our tests, this can be devastating.
Now Dawids Ant test launcher tries to be a lot smarter about all this and caches run times so that it can order tests and holds some back for idle JVM's, etc. So I was looking at already very slow tests times on master often being much worse on gradle depending on if you ran tests across modules (which even the field a bit).
We are a terrible case for gradles parallel tests.

I figured I would improve tests times like usual to solve this as much as I could, but it's really not enough - our project is too large, too many tests, projects much smaller than our complain about this gradle issue.

Luckily though, the abstractions and code that runs tests is super simple. So in the end, I made a new test target called testFast for now, and it's mostly feature complete with the test target (extends the Test task) except that it doesn't support rerunning just failed tests currently and doesn't support looking for tests via reflection, just file inclusions / exclusion regex.

It's really just simple logic changes of a pretty simple process and gives us a lot of options. Of course we will leave gradles default test task in place and this will be an alternate target.

The regular test task will find tests and send every single one in before any ever get processed, so that is changed to start processing right away instead (which is why the retry failed tests processor was taken out of the chain). So testFast scans all the tests and puts them into normal, slow, and slowest categories based on annotations. It shuffles those buckets and sends the slowest to the JVMs first. Then we hold tests back if too many are outstanding so that when we get towards the end, tests can go to idle JVM's instead of lining up early not knowing if it's a very slow line. When the tests hit the JVM's, instead of simple round robin, we look at how many tests are already in line at each JVM and choose the shortest line or shuffle.

So I started with solr-core generally taking 20+ minutes or more on my 18 core machine, and with some other simple test taming, testFail can now do it in just under 7 minutes.

Very easy to add the beasting support that is not already part of the test runner and easy to make sure the test runner can do it's job correctly.

- Mark 
123