Replicate configoverlay.json

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Replicate configoverlay.json

Sundaram, Dinesh

Team,

 

Can you please share the steps to replicate configoverlay.json from Master to Slave… in other words, how do we replicate from Master to Slave if any configuration updated via API.

 

Dinesh Sundaram

MBS Platform Engineering

 

Mastercard

 

CONFIDENTIALITY NOTICE This e-mail message and any attachments are only for the use of the intended recipient and may contain information that is privileged, confidential or exempt from disclosure under applicable law. If you are not the intended recipient, any disclosure, distribution or other use of this e-mail message or attachments is prohibited. If you have received this e-mail message in error, please delete and notify the sender immediately. Thank you.
Reply | Threaded
Open this post in threaded view
|

Re: Replicate configoverlay.json

Shawn Heisey-2
On 3/6/2018 10:50 AM, Sundaram, Dinesh wrote:
> Can you please share the steps to replicate configoverlay.json from
> Master to Slave… in other words, how do we replicate from Master to
> Slave if any configuration updated via API.

If that file is in the same place as solrconfig.xml, then you would add
it to the "confFiles" parameter in the master replication config.  If it
gets saved somewhere else, then I don't know if it would be possible. 
I've never used the config overlay, but it sounds like it probably gets
saved in the conf directory along with the rest of the config files.

https://lucene.apache.org/solr/guide/6_6/index-replication.html#IndexReplication-ConfiguringtheReplicationRequestHandleronaMasterServer

Thanks,
Shawn

Reply | Threaded
Open this post in threaded view
|

RE: Replicate configoverlay.json

Sundaram, Dinesh
Thanks Shawn for checking this. configverlay.json is not available under /conf. Actually it is dynamic file which is available in zookeeper log.1 binary file. So whenever we do the config update via API it will get saved directly in the zookeeper log.1 binary file. Is there any way to replicate to Slaves if any update happens to this file.


--Dinesh Sundaram


-----Original Message-----
From: Shawn Heisey [mailto:[hidden email]]
Sent: Wednesday, March 7, 2018 6:09 PM
To: [hidden email]
Subject: Re: Replicate configoverlay.json

On 3/6/2018 10:50 AM, Sundaram, Dinesh wrote:
> Can you please share the steps to replicate configoverlay.json from
> Master to Slave… in other words, how do we replicate from Master to
> Slave if any configuration updated via API.

If that file is in the same place as solrconfig.xml, then you would add it to the "confFiles" parameter in the master replication config.  If it gets saved somewhere else, then I don't know if it would be possible. I've never used the config overlay, but it sounds like it probably gets saved in the conf directory along with the rest of the config files.

https://urldefense.proofpoint.com/v2/url?u=https-3A__lucene.apache.org_solr_guide_6-5F6_index-2Dreplication.html-23IndexReplication-2DConfiguringtheReplicationRequestHandleronaMasterServer&d=DwIDaQ&c=uc5ZRXl8dGLM1RMQwf7xTCjRqXF0jmCF6SP0bDlmMmY&r=gCFZFMR7y0gzhIBFz1lKTqHFMl-3R6gq7ojE0Eam2Eg&m=2esFnHTWn51-I-gU9Ncq0-q5D1G5Y1n22XYkbfOBJOk&s=tDqgB09Ys3qiifFnM5p1melYpGuo3_HMT7LbYGGjjAE&e=

Thanks,
Shawn


CONFIDENTIALITY NOTICE This e-mail message and any attachments are only for the use of the intended recipient and may contain information that is privileged, confidential or exempt from disclosure under applicable law. If you are not the intended recipient, any disclosure, distribution or other use of this e-mail message or attachments is prohibited. If you have received this e-mail message in error, please delete and notify the sender immediately. Thank you.
Reply | Threaded
Open this post in threaded view
|

Re: Replicate configoverlay.json

Shawn Heisey-2
On 3/8/2018 8:48 AM, Sundaram, Dinesh wrote:
> Thanks Shawn for checking this. configverlay.json is not available under /conf. Actually it is dynamic file which is available in zookeeper log.1 binary file. So whenever we do the config update via API it will get saved directly in the zookeeper log.1 binary file. Is there any way to replicate to Slaves if any update happens to this file.

If you're running zookeeper, then you're running SolrCloud.  And if
you're running SolrCloud, then you cannot (or at least SHOULD NOT) be
using master/slave replication to keep things in sync.

With collections in SolrCloud, any changes to your config with the
config API should take effect on all replicas as quickly as Solr can get
them reloaded.  This is because the configuration is not on disk, it's
in zookeeper, so all replicas should be using exactly the same config.

Thanks,
Shawn

Reply | Threaded
Open this post in threaded view
|

RE: Replicate configoverlay.json

Sundaram, Dinesh
Well. I have mixed cloud and master/slave concepts since solr is supporting. Is there nay way to replicate dynamic confugurations to slave without zookeeper?

--Dinesh Sundaram



-----Original Message-----
From: Shawn Heisey [mailto:[hidden email]]
Sent: Thursday, March 8, 2018 10:23 AM
To: [hidden email]
Subject: Re: Replicate configoverlay.json

On 3/8/2018 8:48 AM, Sundaram, Dinesh wrote:
> Thanks Shawn for checking this. configverlay.json is not available under /conf. Actually it is dynamic file which is available in zookeeper log.1 binary file. So whenever we do the config update via API it will get saved directly in the zookeeper log.1 binary file. Is there any way to replicate to Slaves if any update happens to this file.

If you're running zookeeper, then you're running SolrCloud.  And if you're running SolrCloud, then you cannot (or at least SHOULD NOT) be using master/slave replication to keep things in sync.

With collections in SolrCloud, any changes to your config with the config API should take effect on all replicas as quickly as Solr can get them reloaded.  This is because the configuration is not on disk, it's in zookeeper, so all replicas should be using exactly the same config.

Thanks,
Shawn


CONFIDENTIALITY NOTICE This e-mail message and any attachments are only for the use of the intended recipient and may contain information that is privileged, confidential or exempt from disclosure under applicable law. If you are not the intended recipient, any disclosure, distribution or other use of this e-mail message or attachments is prohibited. If you have received this e-mail message in error, please delete and notify the sender immediately. Thank you.
Reply | Threaded
Open this post in threaded view
|

Re: Replicate configoverlay.json

Shawn Heisey-2
On 3/8/2018 9:25 AM, Sundaram, Dinesh wrote:
> Well. I have mixed cloud and master/slave concepts since solr is supporting. Is there nay way to replicate dynamic confugurations to slave without zookeeper?

If the configoverlay.json file is in the same zookeeper path as
solrconfig.xml, then you MIGHT be able to use confFiles like I described
before.  I have no idea whether it would actually work, though.  It's
entirely possible that SolrCloud cores will not transfer conf files to
slaves.  I've never tried it, I have no idea whether it's going to work
or not.

With SolrCloud, all configuration is in zookeeper.  If a conf directory
exists in a core directory, it is completely ignored.

I will go on record to say that if you're running SolrCloud, you should
not be using master/slave replication. SolrCloud *does* take over the
replication handler for its own purposes, and configures it dynamically
based on the needs of the moment.  I do not know what that would do to a
typical master/slave setup.  If you're going to do it anyway, you should
probably be using a handler with a name other than "/replication" for
your master-slave setup, because that's the handler name that SolrCloud
will use.

Thanks,
Shawn