Build Lucene/Solr in cloud

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Build Lucene/Solr in cloud

Pushkar Raste
Hi,

Building Lucene/Solr with all tests takes about ~50 minutes to an hour depending on how powerful your machine is.
Try out https://crave.io/ to run your builds in the cloud and free up resources on your development machine.

To run the builds in the cloud, just download crave and simply run following command from within your lucene/solr source code directory:
$ <path/to/crave>/crave run ant <target>
Crave will pick up the local changes on your development machine while building in the cloud.

Let me know your experience.

Reply | Threaded
Open this post in threaded view
|

Re: Build Lucene/Solr in cloud

david.w.smiley@gmail.com
Hi,

This is pretty cool!  It worked for me right away without issue.  I have my own similar rsync based script I've been using to build Lucene/Solr on other machines/VM -- https://gist.github.com/dsmiley/fdd589758cd74009222c518640b093b5  It's generous for crave.io to offer free build servers.  However most of my builds I will continue to use my "buildbox" script because I have access to a much beefier machine.

~ David


On Mon, May 18, 2020 at 10:00 AM Pushkar Raste <[hidden email]> wrote:
Hi,

Building Lucene/Solr with all tests takes about ~50 minutes to an hour depending on how powerful your machine is.
Try out https://crave.io/ to run your builds in the cloud and free up resources on your development machine.

To run the builds in the cloud, just download crave and simply run following command from within your lucene/solr source code directory:
$ <path/to/crave>/crave run ant <target>
Crave will pick up the local changes on your development machine while building in the cloud.

Let me know your experience.

Reply | Threaded
Open this post in threaded view
|

Re: Build Lucene/Solr in cloud

Pushkar Raste
Hi David,
Good to hear it worked for you. I suggested Crave team to join the mailing list so they can directly hear the feedback and answer any questions. For rest of us in the community who may not have beefier boxes can you share what is config of your buildbox, how long builds take on the buildbox and if there are any tricks you have to make builds run faster. I don't think Crave supports gradle builds for Lucene/Solr yet but the Crave team can add it if needed (IIUC gradle build is work in progress so not sure how many of us are using it). Crave team will use your feedback to reconfigure the cloud server.

Note: I don't work for Crave but know the founder/CEO and he was generous enough to help set up crave to build Lucene/Solr. I thought others in the community can benefit from it as well. 


On Tue, May 19, 2020 at 11:40 AM David Smiley <[hidden email]> wrote:
Hi,

This is pretty cool!  It worked for me right away without issue.  I have my own similar rsync based script I've been using to build Lucene/Solr on other machines/VM -- https://gist.github.com/dsmiley/fdd589758cd74009222c518640b093b5  It's generous for crave.io to offer free build servers.  However most of my builds I will continue to use my "buildbox" script because I have access to a much beefier machine.

~ David


On Mon, May 18, 2020 at 10:00 AM Pushkar Raste <[hidden email]> wrote:
Hi,

Building Lucene/Solr with all tests takes about ~50 minutes to an hour depending on how powerful your machine is.
Try out https://crave.io/ to run your builds in the cloud and free up resources on your development machine.

To run the builds in the cloud, just download crave and simply run following command from within your lucene/solr source code directory:
$ <path/to/crave>/crave run ant <target>
Crave will pick up the local changes on your development machine while building in the cloud.

Let me know your experience.

Reply | Threaded
Open this post in threaded view
|

Re: Build Lucene/Solr in cloud

Yuvraaj Kelkar
Hi Pushkar,
Thanks for your kind words!

Hi David,
Glad to hear the build was smooth with Crave.  I echo Pushkar's questions.
Also, I am all ears to any other feedback and questions you may have.

Thanks,
-Uv

On May 20 2020, at 5:17 am, Pushkar Raste <[hidden email]> wrote:
Hi David,
Good to hear it worked for you. I suggested Crave team to join the mailing list so they can directly hear the feedback and answer any questions. For rest of us in the community who may not have beefier boxes can you share what is config of your buildbox, how long builds take on the buildbox and if there are any tricks you have to make builds run faster. I don't think Crave supports gradle builds for Lucene/Solr yet but the Crave team can add it if needed (IIUC gradle build is work in progress so not sure how many of us are using it). Crave team will use your feedback to reconfigure the cloud server.

Note: I don't work for Crave but know the founder/CEO and he was generous enough to help set up crave to build Lucene/Solr. I thought others in the community can benefit from it as well. 


Sent from Mailspring
On Tue, May 19, 2020 at 11:40 AM David Smiley <[hidden email]> wrote:
Hi,

This is pretty cool!  It worked for me right away without issue.  I have my own similar rsync based script I've been using to build Lucene/Solr on other machines/VM -- https://gist.github.com/dsmiley/fdd589758cd74009222c518640b093b5  It's generous for crave.io to offer free build servers.  However most of my builds I will continue to use my "buildbox" script because I have access to a much beefier machine.

~ David


On Mon, May 18, 2020 at 10:00 AM Pushkar Raste <[hidden email]> wrote:
Hi,

Building Lucene/Solr with all tests takes about ~50 minutes to an hour depending on how powerful your machine is.
Try out https://crave.io/ to run your builds in the cloud and free up resources on your development machine.

To run the builds in the cloud, just download crave and simply run following command from within your lucene/solr source code directory:
$ <path/to/crave>/crave run ant <target>
Crave will pick up the local changes on your development machine while building in the cloud.

Let me know your experience.
Reply | Threaded
Open this post in threaded view
|

Re: Build Lucene/Solr in cloud

david.w.smiley@gmail.com
Hi Yuvraaj,

I do builds on a corporate provided VM that I don't know a lot about but it reports 16 CPUs and plenty of RAM (32GB?).  I tuned the Solr build to use 10 test runners, which seems to work out best.  Lately, the Solr-only tests take about 21-22 minutes or so.  I run with "ant -f solr/build.xml test"

Since I didn't need to touch Crave's config, I don't have much feedback for it.  It'd be nice if you could pass it a variable number of args that it would run similar to an "ssh" command, etc.  My buildbox.sh script (linked in the gist) works this way.

When I next do dev from my personal laptop, I'll use Crave.io.

~ David

On Wed, May 20, 2020 at 7:39 PM Yuvraaj Kelkar <[hidden email]> wrote:
Hi Pushkar,
Thanks for your kind words!

Hi David,
Glad to hear the build was smooth with Crave.  I echo Pushkar's questions.
Also, I am all ears to any other feedback and questions you may have.

Thanks,
-Uv

On May 20 2020, at 5:17 am, Pushkar Raste <[hidden email]> wrote:
Hi David,
Good to hear it worked for you. I suggested Crave team to join the mailing list so they can directly hear the feedback and answer any questions. For rest of us in the community who may not have beefier boxes can you share what is config of your buildbox, how long builds take on the buildbox and if there are any tricks you have to make builds run faster. I don't think Crave supports gradle builds for Lucene/Solr yet but the Crave team can add it if needed (IIUC gradle build is work in progress so not sure how many of us are using it). Crave team will use your feedback to reconfigure the cloud server.

Note: I don't work for Crave but know the founder/CEO and he was generous enough to help set up crave to build Lucene/Solr. I thought others in the community can benefit from it as well. 


Sent from Mailspring
On Tue, May 19, 2020 at 11:40 AM David Smiley <[hidden email]> wrote:
Hi,

This is pretty cool!  It worked for me right away without issue.  I have my own similar rsync based script I've been using to build Lucene/Solr on other machines/VM -- https://gist.github.com/dsmiley/fdd589758cd74009222c518640b093b5  It's generous for crave.io to offer free build servers.  However most of my builds I will continue to use my "buildbox" script because I have access to a much beefier machine.

~ David


On Mon, May 18, 2020 at 10:00 AM Pushkar Raste <[hidden email]> wrote:
Hi,

Building Lucene/Solr with all tests takes about ~50 minutes to an hour depending on how powerful your machine is.
Try out https://crave.io/ to run your builds in the cloud and free up resources on your development machine.

To run the builds in the cloud, just download crave and simply run following command from within your lucene/solr source code directory:
$ <path/to/crave>/crave run ant <target>
Crave will pick up the local changes on your development machine while building in the cloud.

Let me know your experience.
Reply | Threaded
Open this post in threaded view
|

Re: Build Lucene/Solr in cloud

david.w.smiley@gmail.com
I don't know what Crave.io does should I, say, close my laptop and go to sleep and come back to it.  If it could survive that somehow then that'd be a sweet feature!  I doubt my simple rsync script plays well with that so I don't dare.

~ David


On Wed, May 20, 2020 at 10:05 PM David Smiley <[hidden email]> wrote:
Hi Yuvraaj,

I do builds on a corporate provided VM that I don't know a lot about but it reports 16 CPUs and plenty of RAM (32GB?).  I tuned the Solr build to use 10 test runners, which seems to work out best.  Lately, the Solr-only tests take about 21-22 minutes or so.  I run with "ant -f solr/build.xml test"

Since I didn't need to touch Crave's config, I don't have much feedback for it.  It'd be nice if you could pass it a variable number of args that it would run similar to an "ssh" command, etc.  My buildbox.sh script (linked in the gist) works this way.

When I next do dev from my personal laptop, I'll use Crave.io.

~ David

On Wed, May 20, 2020 at 7:39 PM Yuvraaj Kelkar <[hidden email]> wrote:
Hi Pushkar,
Thanks for your kind words!

Hi David,
Glad to hear the build was smooth with Crave.  I echo Pushkar's questions.
Also, I am all ears to any other feedback and questions you may have.

Thanks,
-Uv

On May 20 2020, at 5:17 am, Pushkar Raste <[hidden email]> wrote:
Hi David,
Good to hear it worked for you. I suggested Crave team to join the mailing list so they can directly hear the feedback and answer any questions. For rest of us in the community who may not have beefier boxes can you share what is config of your buildbox, how long builds take on the buildbox and if there are any tricks you have to make builds run faster. I don't think Crave supports gradle builds for Lucene/Solr yet but the Crave team can add it if needed (IIUC gradle build is work in progress so not sure how many of us are using it). Crave team will use your feedback to reconfigure the cloud server.

Note: I don't work for Crave but know the founder/CEO and he was generous enough to help set up crave to build Lucene/Solr. I thought others in the community can benefit from it as well. 


Sent from Mailspring
On Tue, May 19, 2020 at 11:40 AM David Smiley <[hidden email]> wrote:
Hi,

This is pretty cool!  It worked for me right away without issue.  I have my own similar rsync based script I've been using to build Lucene/Solr on other machines/VM -- https://gist.github.com/dsmiley/fdd589758cd74009222c518640b093b5  It's generous for crave.io to offer free build servers.  However most of my builds I will continue to use my "buildbox" script because I have access to a much beefier machine.

~ David


On Mon, May 18, 2020 at 10:00 AM Pushkar Raste <[hidden email]> wrote:
Hi,

Building Lucene/Solr with all tests takes about ~50 minutes to an hour depending on how powerful your machine is.
Try out https://crave.io/ to run your builds in the cloud and free up resources on your development machine.

To run the builds in the cloud, just download crave and simply run following command from within your lucene/solr source code directory:
$ <path/to/crave>/crave run ant <target>
Crave will pick up the local changes on your development machine while building in the cloud.

Let me know your experience.
Reply | Threaded
Open this post in threaded view
|

Re: Build Lucene/Solr in cloud

Yuvraaj Kelkar
Hi David,

@node spec:
The current setup has Crave configured to spin up a 16 core (VCPU) with 64 GB ram.
I can change that to more cores, more memory if you want to try it out.

@variable number of arguments:
Crave does accept a variable number of arguments. It is very similar to the ssh style of unix commands:
crave run -- command with multiple parameters -with -flags -like -this 

You could even run multiple commands:
crave run 'command1 args1 ; command2 args2' 

@laptop sleep:
There's two ways to survive sleep:
  1. Start it in "detached" mode: Starts the task in the background on the remote node. You don't need a persistent network connection for this. It just runs it in the background. Use: crave run --detached -- command args 
  2. Start Crave in foreground mode, then let the laptop sleep. This will cause the network connection to break and the crave client will terminate. However, the task on the remote end will continue. Client side failures don't stop remote side work.
Note: A Ctrl+C on the crave client side will be transmitted to the remote end and can be used to terminate a foreground task.
To see running tasks, use crave list .
To kill background tasks use crave stop .

Thanks,
-Uv

On May 20 2020, at 7:09 pm, David Smiley <[hidden email]> wrote:
I don't know what Crave.io does should I, say, close my laptop and go to sleep and come back to it.  If it could survive that somehow then that'd be a sweet feature!  I doubt my simple rsync script plays well with that so I don't dare.

~ David


Sent from Mailspring
On Wed, May 20, 2020 at 10:05 PM David Smiley <[hidden email]> wrote:
Hi Yuvraaj,

I do builds on a corporate provided VM that I don't know a lot about but it reports 16 CPUs and plenty of RAM (32GB?).  I tuned the Solr build to use 10 test runners, which seems to work out best.  Lately, the Solr-only tests take about 21-22 minutes or so.  I run with "ant -f solr/build.xml test"

Since I didn't need to touch Crave's config, I don't have much feedback for it.  It'd be nice if you could pass it a variable number of args that it would run similar to an "ssh" command, etc.  My buildbox.sh script (linked in the gist) works this way.

When I next do dev from my personal laptop, I'll use Crave.io.

~ David

On Wed, May 20, 2020 at 7:39 PM Yuvraaj Kelkar <[hidden email]> wrote:
Hi Pushkar,
Thanks for your kind words!

Hi David,
Glad to hear the build was smooth with Crave.  I echo Pushkar's questions.
Also, I am all ears to any other feedback and questions you may have.

Thanks,
-Uv

On May 20 2020, at 5:17 am, Pushkar Raste <[hidden email]> wrote:
Hi David,
Good to hear it worked for you. I suggested Crave team to join the mailing list so they can directly hear the feedback and answer any questions. For rest of us in the community who may not have beefier boxes can you share what is config of your buildbox, how long builds take on the buildbox and if there are any tricks you have to make builds run faster. I don't think Crave supports gradle builds for Lucene/Solr yet but the Crave team can add it if needed (IIUC gradle build is work in progress so not sure how many of us are using it). Crave team will use your feedback to reconfigure the cloud server.

Note: I don't work for Crave but know the founder/CEO and he was generous enough to help set up crave to build Lucene/Solr. I thought others in the community can benefit from it as well. 


On Tue, May 19, 2020 at 11:40 AM David Smiley <[hidden email]> wrote:
Hi,

This is pretty cool!  It worked for me right away without issue.  I have my own similar rsync based script I've been using to build Lucene/Solr on other machines/VM -- https://gist.github.com/dsmiley/fdd589758cd74009222c518640b093b5  It's generous for crave.io to offer free build servers.  However most of my builds I will continue to use my "buildbox" script because I have access to a much beefier machine.

~ David


On Mon, May 18, 2020 at 10:00 AM Pushkar Raste <[hidden email]> wrote:
Hi,

Building Lucene/Solr with all tests takes about ~50 minutes to an hour depending on how powerful your machine is.
Try out https://crave.io/ to run your builds in the cloud and free up resources on your development machine.

To run the builds in the cloud, just download crave and simply run following command from within your lucene/solr source code directory:
$ <path/to/crave>/crave run ant <target>
Crave will pick up the local changes on your development machine while building in the cloud.

Let me know your experience.
Reply | Threaded
Open this post in threaded view
|

Re: Build Lucene/Solr in cloud

david.w.smiley@gmail.com
Aha, thanks!

It appear's the build's CPU detection is failing to see how many are actually available in the Crave environment.  So I am manually upping it and I'm seeing much better build times, at least on the Lucene end.  I'll now let this run overnight for the Solr side:

crave run -- ant -f solr/build.xml test -Dtests.jvms=10


And put my computer to sleep too and I'll see if I can get at the results easily.

~ David


On Thu, May 21, 2020 at 12:32 AM Yuvraaj Kelkar <[hidden email]> wrote:
Hi David,

@node spec:
The current setup has Crave configured to spin up a 16 core (VCPU) with 64 GB ram.
I can change that to more cores, more memory if you want to try it out.

@variable number of arguments:
Crave does accept a variable number of arguments. It is very similar to the ssh style of unix commands:
crave run -- command with multiple parameters -with -flags -like -this 

You could even run multiple commands:
crave run 'command1 args1 ; command2 args2' 

@laptop sleep:
There's two ways to survive sleep:
  1. Start it in "detached" mode: Starts the task in the background on the remote node. You don't need a persistent network connection for this. It just runs it in the background. Use: crave run --detached -- command args 
  2. Start Crave in foreground mode, then let the laptop sleep. This will cause the network connection to break and the crave client will terminate. However, the task on the remote end will continue. Client side failures don't stop remote side work.
Note: A Ctrl+C on the crave client side will be transmitted to the remote end and can be used to terminate a foreground task.
To see running tasks, use crave list .
To kill background tasks use crave stop .

Thanks,
-Uv

On May 20 2020, at 7:09 pm, David Smiley <[hidden email]> wrote:
I don't know what Crave.io does should I, say, close my laptop and go to sleep and come back to it.  If it could survive that somehow then that'd be a sweet feature!  I doubt my simple rsync script plays well with that so I don't dare.

~ David


Sent from Mailspring
On Wed, May 20, 2020 at 10:05 PM David Smiley <[hidden email]> wrote:
Hi Yuvraaj,

I do builds on a corporate provided VM that I don't know a lot about but it reports 16 CPUs and plenty of RAM (32GB?).  I tuned the Solr build to use 10 test runners, which seems to work out best.  Lately, the Solr-only tests take about 21-22 minutes or so.  I run with "ant -f solr/build.xml test"

Since I didn't need to touch Crave's config, I don't have much feedback for it.  It'd be nice if you could pass it a variable number of args that it would run similar to an "ssh" command, etc.  My buildbox.sh script (linked in the gist) works this way.

When I next do dev from my personal laptop, I'll use Crave.io.

~ David

On Wed, May 20, 2020 at 7:39 PM Yuvraaj Kelkar <[hidden email]> wrote:
Hi Pushkar,
Thanks for your kind words!

Hi David,
Glad to hear the build was smooth with Crave.  I echo Pushkar's questions.
Also, I am all ears to any other feedback and questions you may have.

Thanks,
-Uv

On May 20 2020, at 5:17 am, Pushkar Raste <[hidden email]> wrote:
Hi David,
Good to hear it worked for you. I suggested Crave team to join the mailing list so they can directly hear the feedback and answer any questions. For rest of us in the community who may not have beefier boxes can you share what is config of your buildbox, how long builds take on the buildbox and if there are any tricks you have to make builds run faster. I don't think Crave supports gradle builds for Lucene/Solr yet but the Crave team can add it if needed (IIUC gradle build is work in progress so not sure how many of us are using it). Crave team will use your feedback to reconfigure the cloud server.

Note: I don't work for Crave but know the founder/CEO and he was generous enough to help set up crave to build Lucene/Solr. I thought others in the community can benefit from it as well. 


On Tue, May 19, 2020 at 11:40 AM David Smiley <[hidden email]> wrote:
Hi,

This is pretty cool!  It worked for me right away without issue.  I have my own similar rsync based script I've been using to build Lucene/Solr on other machines/VM -- https://gist.github.com/dsmiley/fdd589758cd74009222c518640b093b5  It's generous for crave.io to offer free build servers.  However most of my builds I will continue to use my "buildbox" script because I have access to a much beefier machine.

~ David


On Mon, May 18, 2020 at 10:00 AM Pushkar Raste <[hidden email]> wrote:
Hi,

Building Lucene/Solr with all tests takes about ~50 minutes to an hour depending on how powerful your machine is.
Try out https://crave.io/ to run your builds in the cloud and free up resources on your development machine.

To run the builds in the cloud, just download crave and simply run following command from within your lucene/solr source code directory:
$ <path/to/crave>/crave run ant <target>
Crave will pick up the local changes on your development machine while building in the cloud.

Let me know your experience.
Reply | Threaded
Open this post in threaded view
|

Re: Build Lucene/Solr in cloud

david.w.smiley@gmail.com
It took 26min; not bad!  Well of course I wish it took much less time but I'm highly appreciative of donated build resources!

I also tried when my computer went to sleep.  It seems I'm unable to retrieve info about the build because I need a Crave login. Is that possible with free/donated builds?

~ David


On Thu, May 21, 2020 at 2:21 AM David Smiley <[hidden email]> wrote:
Aha, thanks!

It appear's the build's CPU detection is failing to see how many are actually available in the Crave environment.  So I am manually upping it and I'm seeing much better build times, at least on the Lucene end.  I'll now let this run overnight for the Solr side:

crave run -- ant -f solr/build.xml test -Dtests.jvms=10


And put my computer to sleep too and I'll see if I can get at the results easily.

~ David


On Thu, May 21, 2020 at 12:32 AM Yuvraaj Kelkar <[hidden email]> wrote:
Hi David,

@node spec:
The current setup has Crave configured to spin up a 16 core (VCPU) with 64 GB ram.
I can change that to more cores, more memory if you want to try it out.

@variable number of arguments:
Crave does accept a variable number of arguments. It is very similar to the ssh style of unix commands:
crave run -- command with multiple parameters -with -flags -like -this 

You could even run multiple commands:
crave run 'command1 args1 ; command2 args2' 

@laptop sleep:
There's two ways to survive sleep:
  1. Start it in "detached" mode: Starts the task in the background on the remote node. You don't need a persistent network connection for this. It just runs it in the background. Use: crave run --detached -- command args 
  2. Start Crave in foreground mode, then let the laptop sleep. This will cause the network connection to break and the crave client will terminate. However, the task on the remote end will continue. Client side failures don't stop remote side work.
Note: A Ctrl+C on the crave client side will be transmitted to the remote end and can be used to terminate a foreground task.
To see running tasks, use crave list .
To kill background tasks use crave stop .

Thanks,
-Uv

On May 20 2020, at 7:09 pm, David Smiley <[hidden email]> wrote:
I don't know what Crave.io does should I, say, close my laptop and go to sleep and come back to it.  If it could survive that somehow then that'd be a sweet feature!  I doubt my simple rsync script plays well with that so I don't dare.

~ David


Sent from Mailspring
On Wed, May 20, 2020 at 10:05 PM David Smiley <[hidden email]> wrote:
Hi Yuvraaj,

I do builds on a corporate provided VM that I don't know a lot about but it reports 16 CPUs and plenty of RAM (32GB?).  I tuned the Solr build to use 10 test runners, which seems to work out best.  Lately, the Solr-only tests take about 21-22 minutes or so.  I run with "ant -f solr/build.xml test"

Since I didn't need to touch Crave's config, I don't have much feedback for it.  It'd be nice if you could pass it a variable number of args that it would run similar to an "ssh" command, etc.  My buildbox.sh script (linked in the gist) works this way.

When I next do dev from my personal laptop, I'll use Crave.io.

~ David

On Wed, May 20, 2020 at 7:39 PM Yuvraaj Kelkar <[hidden email]> wrote:
Hi Pushkar,
Thanks for your kind words!

Hi David,
Glad to hear the build was smooth with Crave.  I echo Pushkar's questions.
Also, I am all ears to any other feedback and questions you may have.

Thanks,
-Uv

On May 20 2020, at 5:17 am, Pushkar Raste <[hidden email]> wrote:
Hi David,
Good to hear it worked for you. I suggested Crave team to join the mailing list so they can directly hear the feedback and answer any questions. For rest of us in the community who may not have beefier boxes can you share what is config of your buildbox, how long builds take on the buildbox and if there are any tricks you have to make builds run faster. I don't think Crave supports gradle builds for Lucene/Solr yet but the Crave team can add it if needed (IIUC gradle build is work in progress so not sure how many of us are using it). Crave team will use your feedback to reconfigure the cloud server.

Note: I don't work for Crave but know the founder/CEO and he was generous enough to help set up crave to build Lucene/Solr. I thought others in the community can benefit from it as well. 


On Tue, May 19, 2020 at 11:40 AM David Smiley <[hidden email]> wrote:
Hi,

This is pretty cool!  It worked for me right away without issue.  I have my own similar rsync based script I've been using to build Lucene/Solr on other machines/VM -- https://gist.github.com/dsmiley/fdd589758cd74009222c518640b093b5  It's generous for crave.io to offer free build servers.  However most of my builds I will continue to use my "buildbox" script because I have access to a much beefier machine.

~ David


On Mon, May 18, 2020 at 10:00 AM Pushkar Raste <[hidden email]> wrote:
Hi,

Building Lucene/Solr with all tests takes about ~50 minutes to an hour depending on how powerful your machine is.
Try out https://crave.io/ to run your builds in the cloud and free up resources on your development machine.

To run the builds in the cloud, just download crave and simply run following command from within your lucene/solr source code directory:
$ <path/to/crave>/crave run ant <target>
Crave will pick up the local changes on your development machine while building in the cloud.

Let me know your experience.
Reply | Threaded
Open this post in threaded view
|

Re: Build Lucene/Solr in cloud

Yuvraaj Kelkar
@speed:
I haven't delved into the actual ant build files, but from a cursory look at the CPU utilization during an ant task, should I assume that more cores = faster?
If there's other information that you can share about how to make it faster, or if you have future plans in that direction, please tell me so I can share with the team.
We're always looking to improve the underlying platform.

@login:
The anonymous login method has no real account tied to it, so there's no way right now to log into it.
Ideally every user gets a non-anon account, but that's typically something that corps ask for.

@stdout/stderr:
That said, we're figuring out a way to retrieve postmortem, the stdout/stderr of your build even on anon usage.

@gradle requirements
With a bit of trial and error and the Solr readme.md I was able to get the current Ant environment up a few days ago:
https://github.com/accupara/docker-images/blob/master/java/solr/Dockerfile
After looking at the mailing list archive for a bit, I see that the community is trying to get a working Gradle build.
Can you point me to any docs for the Solr specific Gradle requirements so I can enable that in the Solr image we've created?

Thanks,
-Uv
On May 21 2020, at 8:53 am, David Smiley <[hidden email]> wrote:
It took 26min; not bad!  Well of course I wish it took much less time but I'm highly appreciative of donated build resources!

I also tried when my computer went to sleep.  It seems I'm unable to retrieve info about the build because I need a Crave login. Is that possible with free/donated builds?

~ David


Sent from Mailspring
On Thu, May 21, 2020 at 2:21 AM David Smiley <[hidden email]> wrote:
Aha, thanks!

It appear's the build's CPU detection is failing to see how many are actually available in the Crave environment.  So I am manually upping it and I'm seeing much better build times, at least on the Lucene end.  I'll now let this run overnight for the Solr side:

crave run -- ant -f solr/build.xml test -Dtests.jvms=10

And put my computer to sleep too and I'll see if I can get at the results easily.

~ David


On Thu, May 21, 2020 at 12:32 AM Yuvraaj Kelkar <[hidden email]> wrote:
Hi David,

@node spec:
The current setup has Crave configured to spin up a 16 core (VCPU) with 64 GB ram.
I can change that to more cores, more memory if you want to try it out.

@variable number of arguments:
Crave does accept a variable number of arguments. It is very similar to the ssh style of unix commands:
crave run -- command with multiple parameters -with -flags -like -this 

You could even run multiple commands:
crave run 'command1 args1 ; command2 args2' 

@laptop sleep:
There's two ways to survive sleep:
  1. Start it in "detached" mode: Starts the task in the background on the remote node. You don't need a persistent network connection for this. It just runs it in the background. Use: crave run --detached -- command args 
  2. Start Crave in foreground mode, then let the laptop sleep. This will cause the network connection to break and the crave client will terminate. However, the task on the remote end will continue. Client side failures don't stop remote side work.
Note: A Ctrl+C on the crave client side will be transmitted to the remote end and can be used to terminate a foreground task.
To see running tasks, use crave list .
To kill background tasks use crave stop .

Thanks,
-Uv

On May 20 2020, at 7:09 pm, David Smiley <[hidden email]> wrote:
I don't know what Crave.io does should I, say, close my laptop and go to sleep and come back to it.  If it could survive that somehow then that'd be a sweet feature!  I doubt my simple rsync script plays well with that so I don't dare.

~ David


On Wed, May 20, 2020 at 10:05 PM David Smiley <[hidden email]> wrote:
Hi Yuvraaj,

I do builds on a corporate provided VM that I don't know a lot about but it reports 16 CPUs and plenty of RAM (32GB?).  I tuned the Solr build to use 10 test runners, which seems to work out best.  Lately, the Solr-only tests take about 21-22 minutes or so.  I run with "ant -f solr/build.xml test"

Since I didn't need to touch Crave's config, I don't have much feedback for it.  It'd be nice if you could pass it a variable number of args that it would run similar to an "ssh" command, etc.  My buildbox.sh script (linked in the gist) works this way.

When I next do dev from my personal laptop, I'll use Crave.io.

~ David

On Wed, May 20, 2020 at 7:39 PM Yuvraaj Kelkar <[hidden email]> wrote:
Hi Pushkar,
Thanks for your kind words!

Hi David,
Glad to hear the build was smooth with Crave.  I echo Pushkar's questions.
Also, I am all ears to any other feedback and questions you may have.

Thanks,
-Uv

On May 20 2020, at 5:17 am, Pushkar Raste <[hidden email]> wrote:
Hi David,
Good to hear it worked for you. I suggested Crave team to join the mailing list so they can directly hear the feedback and answer any questions. For rest of us in the community who may not have beefier boxes can you share what is config of your buildbox, how long builds take on the buildbox and if there are any tricks you have to make builds run faster. I don't think Crave supports gradle builds for Lucene/Solr yet but the Crave team can add it if needed (IIUC gradle build is work in progress so not sure how many of us are using it). Crave team will use your feedback to reconfigure the cloud server.

Note: I don't work for Crave but know the founder/CEO and he was generous enough to help set up crave to build Lucene/Solr. I thought others in the community can benefit from it as well. 


On Tue, May 19, 2020 at 11:40 AM David Smiley <[hidden email]> wrote:
Hi,

This is pretty cool!  It worked for me right away without issue.  I have my own similar rsync based script I've been using to build Lucene/Solr on other machines/VM -- https://gist.github.com/dsmiley/fdd589758cd74009222c518640b093b5  It's generous for crave.io to offer free build servers.  However most of my builds I will continue to use my "buildbox" script because I have access to a much beefier machine.

~ David


On Mon, May 18, 2020 at 10:00 AM Pushkar Raste <[hidden email]> wrote:
Hi,

Building Lucene/Solr with all tests takes about ~50 minutes to an hour depending on how powerful your machine is.
Try out https://crave.io/ to run your builds in the cloud and free up resources on your development machine.

To run the builds in the cloud, just download crave and simply run following command from within your lucene/solr source code directory:
$ <path/to/crave>/crave run ant <target>
Crave will pick up the local changes on your development machine while building in the cloud.

Let me know your experience.
Reply | Threaded
Open this post in threaded view
|

Re: Build Lucene/Solr in cloud

david.w.smiley@gmail.com
On Thu, May 21, 2020 at 2:32 PM Yuvraaj Kelkar <[hidden email]> wrote:
@speed:
I haven't delved into the actual ant build files, but from a cursory look at the CPU utilization during an ant task, should I assume that more cores = faster?
If there's other information that you can share about how to make it faster, or if you have future plans in that direction, please tell me so I can share with the team.
We're always looking to improve the underlying platform.

It's not so much Ant or Gradle as it its having test runners to run tests in parallel.  So yes; more CPUs equals faster build (due to the tests).  But the build may not detect the number of CPUs properly in Crave, from what I see.  It is, at least, explicitly configurable, as I showed.
 
@gradle requirements
With a bit of trial and error and the Solr readme.md I was able to get the current Ant environment up a few days ago:
https://github.com/accupara/docker-images/blob/master/java/solr/Dockerfile
After looking at the mailing list archive for a bit, I see that the community is trying to get a working Gradle build.
Can you point me to any docs for the Solr specific Gradle requirements so I can enable that in the Solr image we've created?

The top level "help/" is pretty good.  See:
and some other sibling files.  Also, I think this GitHub PR hook is useful to glean how simple it is:

Thanks,

 ~ David