Solr cloud - poweroff procedure

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Solr cloud - poweroff procedure

lstusr 5u93n4
Hi All,

We have a solr cloud running 3 shards, 3 hosts, 6 total NRT replicas, and
the data director on hdfs. It has 950 million documents in the index,
occupying 700GB of disk space.

We need to completely power off the system to move it.

Are there any actions we should take on shutdown to help the process?
Anyhing we should expect on power on?

Thanks,

Kyle
Reply | Threaded
Open this post in threaded view
|

Re: Solr cloud - poweroff procedure

Erick Erickson
bin/solr stop

As long as you don't kill it with extreme prejudice (i.e. kill -9 or
pull the plug) it should be fine. Assuming you're running ZooKeeper
in an external ensemble, I'd certainly stop those after all the Solr
instances were stopped.

Powering the nodes up is irrelevant to Solr, the bin/solr start bit
will actually start the Solr instance. And no, there's nothing special
that needs to happen there, it's just like any other time the script
starts Solr. You'll have to have the ZooKeeper instances running for
Solr to come up of course.

Best,
Erick
On Tue, Oct 30, 2018 at 2:57 PM lstusr 5u93n4 <[hidden email]> wrote:

>
> Hi All,
>
> We have a solr cloud running 3 shards, 3 hosts, 6 total NRT replicas, and
> the data director on hdfs. It has 950 million documents in the index,
> occupying 700GB of disk space.
>
> We need to completely power off the system to move it.
>
> Are there any actions we should take on shutdown to help the process?
> Anyhing we should expect on power on?
>
> Thanks,
>
> Kyle
Reply | Threaded
Open this post in threaded view
|

Re: Solr cloud - poweroff procedure

Walter Underwood
I agree.

1. Shut down each Solr server process using the “bin/solr” script.
2. Shut down the Zookeeper ensemble.
3. Take backups.
4. Shut down the OS.

Do that in reverse to get going.

wunder
Walter Underwood
[hidden email]
http://observer.wunderwood.org/  (my blog)

> On Oct 30, 2018, at 8:03 PM, Erick Erickson <[hidden email]> wrote:
>
> bin/solr stop
>
> As long as you don't kill it with extreme prejudice (i.e. kill -9 or
> pull the plug) it should be fine. Assuming you're running ZooKeeper
> in an external ensemble, I'd certainly stop those after all the Solr
> instances were stopped.
>
> Powering the nodes up is irrelevant to Solr, the bin/solr start bit
> will actually start the Solr instance. And no, there's nothing special
> that needs to happen there, it's just like any other time the script
> starts Solr. You'll have to have the ZooKeeper instances running for
> Solr to come up of course.
>
> Best,
> Erick
> On Tue, Oct 30, 2018 at 2:57 PM lstusr 5u93n4 <[hidden email]> wrote:
>>
>> Hi All,
>>
>> We have a solr cloud running 3 shards, 3 hosts, 6 total NRT replicas, and
>> the data director on hdfs. It has 950 million documents in the index,
>> occupying 700GB of disk space.
>>
>> We need to completely power off the system to move it.
>>
>> Are there any actions we should take on shutdown to help the process?
>> Anyhing we should expect on power on?
>>
>> Thanks,
>>
>> Kyle

Reply | Threaded
Open this post in threaded view
|

Re: Solr cloud - poweroff procedure

Shalin Shekhar Mangar
In reply to this post by lstusr 5u93n4
In case you are using a recent Solr 7.x version with collections that have
autoAddReplicas=true, you should disable the auto add replicas feature
before powering off so that Solr does not decide to move replicas around
because nodes have been lost. See
https://lucene.apache.org/solr/guide/7_4/solrcloud-autoscaling-auto-add-replicas.html#using-cluster-property-to-enable-autoaddreplicas

On Wed, Oct 31, 2018 at 3:27 AM lstusr 5u93n4 <[hidden email]> wrote:

> Hi All,
>
> We have a solr cloud running 3 shards, 3 hosts, 6 total NRT replicas, and
> the data director on hdfs. It has 950 million documents in the index,
> occupying 700GB of disk space.
>
> We need to completely power off the system to move it.
>
> Are there any actions we should take on shutdown to help the process?
> Anyhing we should expect on power on?
>
> Thanks,
>
> Kyle
>


--
Regards,
Shalin Shekhar Mangar.
Reply | Threaded
Open this post in threaded view
|

Re: Solr cloud - poweroff procedure

lstusr 5u93n4
Hi,

Yes, zookeeper is external, and yes, we'll definitely wait until after solr
has stopped to bring it down.

Thanks for the tip about disabling `autoAddReplicas`, we definitely don't
want the shards moving around during the process.

Wunder, your point 3 mentions "take backups". Given that our data is on
hdfs (not co-located with the solr servers) and backed up separately, what
else would you recommend backing up?  The contents of the `solr.home.home`
folder seem like good candidates... anything else? Let's say one of the
servers gets dropped during the move, is it sufficient to restore the
contents of `solr.home.home` onto a new server with the same
hostname/solrVersion/zookeeperConfig and bring it up in the same way as the
others?

Thanks all,

Kyle


On Wed, 31 Oct 2018 at 05:22, Shalin Shekhar Mangar <[hidden email]>
wrote:

> In case you are using a recent Solr 7.x version with collections that have
> autoAddReplicas=true, you should disable the auto add replicas feature
> before powering off so that Solr does not decide to move replicas around
> because nodes have been lost. See
>
> https://lucene.apache.org/solr/guide/7_4/solrcloud-autoscaling-auto-add-replicas.html#using-cluster-property-to-enable-autoaddreplicas
>
> On Wed, Oct 31, 2018 at 3:27 AM lstusr 5u93n4 <[hidden email]> wrote:
>
> > Hi All,
> >
> > We have a solr cloud running 3 shards, 3 hosts, 6 total NRT replicas, and
> > the data director on hdfs. It has 950 million documents in the index,
> > occupying 700GB of disk space.
> >
> > We need to completely power off the system to move it.
> >
> > Are there any actions we should take on shutdown to help the process?
> > Anyhing we should expect on power on?
> >
> > Thanks,
> >
> > Kyle
> >
>
>
> --
> Regards,
> Shalin Shekhar Mangar.
>
Reply | Threaded
Open this post in threaded view
|

Re: Solr cloud - poweroff procedure

Walter Underwood
“Take backups” is whatever you need for your environment. In AWS, we snapshot the EBS volumes, and so on.

Backing up the Solr install and home directories would be good. There are some core.properties files in there that seem to be useful. Honestly, I don’t have a complete handle on the details of naming and properties for cores in Solr Cloud.

For the Zookeeper ensemble, remember that each host has a different value in the myid file.

wunder
Walter Underwood
[hidden email]
http://observer.wunderwood.org/  (my blog)

> On Oct 31, 2018, at 5:17 AM, lstusr 5u93n4 <[hidden email]> wrote:
>
> Hi,
>
> Yes, zookeeper is external, and yes, we'll definitely wait until after solr
> has stopped to bring it down.
>
> Thanks for the tip about disabling `autoAddReplicas`, we definitely don't
> want the shards moving around during the process.
>
> Wunder, your point 3 mentions "take backups". Given that our data is on
> hdfs (not co-located with the solr servers) and backed up separately, what
> else would you recommend backing up?  The contents of the `solr.home.home`
> folder seem like good candidates... anything else? Let's say one of the
> servers gets dropped during the move, is it sufficient to restore the
> contents of `solr.home.home` onto a new server with the same
> hostname/solrVersion/zookeeperConfig and bring it up in the same way as the
> others?
>
> Thanks all,
>
> Kyle
>
>
> On Wed, 31 Oct 2018 at 05:22, Shalin Shekhar Mangar <[hidden email]>
> wrote:
>
>> In case you are using a recent Solr 7.x version with collections that have
>> autoAddReplicas=true, you should disable the auto add replicas feature
>> before powering off so that Solr does not decide to move replicas around
>> because nodes have been lost. See
>>
>> https://lucene.apache.org/solr/guide/7_4/solrcloud-autoscaling-auto-add-replicas.html#using-cluster-property-to-enable-autoaddreplicas
>>
>> On Wed, Oct 31, 2018 at 3:27 AM lstusr 5u93n4 <[hidden email]> wrote:
>>
>>> Hi All,
>>>
>>> We have a solr cloud running 3 shards, 3 hosts, 6 total NRT replicas, and
>>> the data director on hdfs. It has 950 million documents in the index,
>>> occupying 700GB of disk space.
>>>
>>> We need to completely power off the system to move it.
>>>
>>> Are there any actions we should take on shutdown to help the process?
>>> Anyhing we should expect on power on?
>>>
>>> Thanks,
>>>
>>> Kyle
>>>
>>
>>
>> --
>> Regards,
>> Shalin Shekhar Mangar.
>>