Copy index from lower environment to Production | SolrCloud

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Copy index from lower environment to Production | SolrCloud

rohankasat
Hi,

In our setup we have two SolrCloud environments running Solr 7.5 version.
Specific to the question - We have one collection with 3 shards and 3
replicas on the lower environment and a newly created mirrored collection
on Production.
Wanted to know on approaches to copy the index for collection from cloud on
lower environment to Production solrcloud instance ?

--

*Regards,Rohan Kasat*
Reply | Threaded
Open this post in threaded view
|

Re: Copy index from lower environment to Production | SolrCloud

Erick Erickson

(t’s usually far easier to create a new collection in your upper environment and index to _that_. Once the indexing is done, use the Collections API CREATEALIAS command to point traffic to the new collection. You can then use the old one to index to and use CREATEALIAS to point to that one, switching back and forth as necessary.


If that’s not possible, try is the low-level replication API call. WARNING, “this should work”, but I haven’t tried it.…..

See “HTTP  API Commands for the Replication Handler” here: https://lucene.apache.org/solr/guide/7_6/index-replication.html

You do _not_ need to set up replication in solrconfig.xml or the like. You want to use “fetchIndex”. Tricks:

1> you must issue this command _to the core_ you want the index copied to. This will be something like “http://blahlbah/solr/collection1_shard1_replica_n2”. Specify the masterURL as the core you want to copy _from_, i.e. your lower environment.

2> Have one and only one replica for each shard running in your higher environment. This will be “leader only”, i.e. no followers. Once you’ve successfully moved the index, bring up any other replicas (or use ADDREPLICA). When they come up they’ll replicate down from the leader.

3> Be very, very careful that you copy the indexes from exactly the same shard, as defined by the range in state.json associated with each shard.

An alternative is to shut down both systems and just copy the data dir (tlog and index) files manually from the lower to upper. You’d need your upper collection to be completely shut down. Again, copy the files to one replica of each shard then bring _only_ that replica up. Once it’s decided to be the leader, bring the other replicas up or ADDREPLICA, whichever is easier.


> On Jun 4, 2019, at 11:36 AM, Rohan Kasat <[hidden email]> wrote:
>
> Hi,
>
> In our setup we have two SolrCloud environments running Solr 7.5 version.
> Specific to the question - We have one collection with 3 shards and 3
> replicas on the lower environment and a newly created mirrored collection
> on Production.
> Wanted to know on approaches to copy the index for collection from cloud on
> lower environment to Production solrcloud instance ?
>
> --
>
> *Regards,Rohan Kasat*

Reply | Threaded
Open this post in threaded view
|

Re: Copy index from lower environment to Production | SolrCloud

rohankasat
Thank you so much Erick.
Will check these out.

Regards,
Rohan Kasat

On Tue, Jun 4, 2019 at 12:54 PM Erick Erickson <[hidden email]>
wrote:

>
> (t’s usually far easier to create a new collection in your upper
> environment and index to _that_. Once the indexing is done, use the
> Collections API CREATEALIAS command to point traffic to the new collection.
> You can then use the old one to index to and use CREATEALIAS to point to
> that one, switching back and forth as necessary.
>
>
> If that’s not possible, try is the low-level replication API call.
> WARNING, “this should work”, but I haven’t tried it.…..
>
> See “HTTP  API Commands for the Replication Handler” here:
> https://lucene.apache.org/solr/guide/7_6/index-replication.html
>
> You do _not_ need to set up replication in solrconfig.xml or the like. You
> want to use “fetchIndex”. Tricks:
>
> 1> you must issue this command _to the core_ you want the index copied to.
> This will be something like “
> http://blahlbah/solr/collection1_shard1_replica_n2”. Specify the
> masterURL as the core you want to copy _from_, i.e. your lower environment.
>
> 2> Have one and only one replica for each shard running in your higher
> environment. This will be “leader only”, i.e. no followers. Once you’ve
> successfully moved the index, bring up any other replicas (or use
> ADDREPLICA). When they come up they’ll replicate down from the leader.
>
> 3> Be very, very careful that you copy the indexes from exactly the same
> shard, as defined by the range in state.json associated with each shard.
>
> An alternative is to shut down both systems and just copy the data dir
> (tlog and index) files manually from the lower to upper. You’d need your
> upper collection to be completely shut down. Again, copy the files to one
> replica of each shard then bring _only_ that replica up. Once it’s decided
> to be the leader, bring the other replicas up or ADDREPLICA, whichever is
> easier.
>
>
> > On Jun 4, 2019, at 11:36 AM, Rohan Kasat <[hidden email]> wrote:
> >
> > Hi,
> >
> > In our setup we have two SolrCloud environments running Solr 7.5 version.
> > Specific to the question - We have one collection with 3 shards and 3
> > replicas on the lower environment and a newly created mirrored collection
> > on Production.
> > Wanted to know on approaches to copy the index for collection from cloud
> on
> > lower environment to Production solrcloud instance ?
> >
> > --
> >
> > *Regards,Rohan Kasat*
>
> --

*Regards,Rohan Kasat*