Writing config directly to zookeeper

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Writing config directly to zookeeper

Aristedes Maniatis-3
Is there any difference between using the tools supplied with Solr to
write configuration to Zookeeper or just writing directly to our
Zookeeper cluster?

We have tooling that makes it much easier to write directly to ZK rather
than having to use yet another tool to do it.

Thanks

Ari

Reply | Threaded
Open this post in threaded view
|

Re: Writing config directly to zookeeper

Shawn Heisey-2
On 4/17/2018 8:54 PM, Aristedes Maniatis wrote:
> Is there any difference between using the tools supplied with Solr to
> write configuration to Zookeeper or just writing directly to our
> Zookeeper cluster?
>
> We have tooling that makes it much easier to write directly to ZK
> rather than having to use yet another tool to do it.

As long as it ends up in the correct path in the ZK structure, it
doesn't matter how it gets there.

The /configs/XXXX location (where XXXX is the config name) should have
the same contents that would normally be found in a conf directory if it
were standalone Solr and not using the standalone configsets feature.

Thanks,
Shawn

Reply | Threaded
Open this post in threaded view
|

Re: Writing config directly to zookeeper

Walter Underwood
I wrote a Python tool to do this. I use the kazoo package to talk to Zookeeper. It starts with the load balancer URL to Solr.

1. Get cluster status.
2. Parse out the Zookeeper config string including chroot.
3. Connect to Zookeeper.
4. Copy the config to the location described in Shawn’s message.
5. Send linkconfig command to the cluster, just to be sure.
6. Reload the collection with an async command.
7. Ping the cluster until the reload is successful on every node.
8. Optionally, rebuild the suggester on each node.

The actual location of the config in Zookeeper is undocumented, as far as I could tell. I used the Solr ZK CLI, then reverse engineered where it put stuff.

The docs need a “Zookeeper file organization” chapter with this info.

Also, it would be nice if the ZKHOST info was available pre-parsed in cluster status.

wunder
Walter Underwood
[hidden email]
http://observer.wunderwood.org/  (my blog)

> On Apr 17, 2018, at 8:20 PM, Shawn Heisey <[hidden email]> wrote:
>
> On 4/17/2018 8:54 PM, Aristedes Maniatis wrote:
>> Is there any difference between using the tools supplied with Solr to write configuration to Zookeeper or just writing directly to our Zookeeper cluster?
>>
>> We have tooling that makes it much easier to write directly to ZK rather than having to use yet another tool to do it.
>
> As long as it ends up in the correct path in the ZK structure, it doesn't matter how it gets there.
>
> The /configs/XXXX location (where XXXX is the config name) should have the same contents that would normally be found in a conf directory if it were standalone Solr and not using the standalone configsets feature.
>
> Thanks,
> Shawn
>

Reply | Threaded
Open this post in threaded view
|

Re: Writing config directly to zookeeper

Arturas Mazeika
Hi Walter,

Thanks for the message. Would you care to share the tool with us? I would
be interested.. Or have you shared it already?

Cheers,
Arturas

On Wed, Apr 18, 2018 at 5:09 PM, Walter Underwood <[hidden email]>
wrote:

> I wrote a Python tool to do this. I use the kazoo package to talk to
> Zookeeper. It starts with the load balancer URL to Solr.
>
> 1. Get cluster status.
> 2. Parse out the Zookeeper config string including chroot.
> 3. Connect to Zookeeper.
> 4. Copy the config to the location described in Shawn’s message.
> 5. Send linkconfig command to the cluster, just to be sure.
> 6. Reload the collection with an async command.
> 7. Ping the cluster until the reload is successful on every node.
> 8. Optionally, rebuild the suggester on each node.
>
> The actual location of the config in Zookeeper is undocumented, as far as
> I could tell. I used the Solr ZK CLI, then reverse engineered where it put
> stuff.
>
> The docs need a “Zookeeper file organization” chapter with this info.
>
> Also, it would be nice if the ZKHOST info was available pre-parsed in
> cluster status.
>
> wunder
> Walter Underwood
> [hidden email]
> http://observer.wunderwood.org/  (my blog)
>
> > On Apr 17, 2018, at 8:20 PM, Shawn Heisey <[hidden email]> wrote:
> >
> > On 4/17/2018 8:54 PM, Aristedes Maniatis wrote:
> >> Is there any difference between using the tools supplied with Solr to
> write configuration to Zookeeper or just writing directly to our Zookeeper
> cluster?
> >>
> >> We have tooling that makes it much easier to write directly to ZK
> rather than having to use yet another tool to do it.
> >
> > As long as it ends up in the correct path in the ZK structure, it
> doesn't matter how it gets there.
> >
> > The /configs/XXXX location (where XXXX is the config name) should have
> the same contents that would normally be found in a conf directory if it
> were standalone Solr and not using the standalone configsets feature.
> >
> > Thanks,
> > Shawn
> >
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Writing config directly to zookeeper

Erick Erickson
There are some perhaps easier ways to manipulate ZK in the "bin/solr"
script if you haven't seen it

bin/solr zk -help

Best,
Erick

On Wed, Apr 18, 2018 at 8:30 AM, Arturas Mazeika <[hidden email]> wrote:

> Hi Walter,
>
> Thanks for the message. Would you care to share the tool with us? I would
> be interested.. Or have you shared it already?
>
> Cheers,
> Arturas
>
> On Wed, Apr 18, 2018 at 5:09 PM, Walter Underwood <[hidden email]>
> wrote:
>
>> I wrote a Python tool to do this. I use the kazoo package to talk to
>> Zookeeper. It starts with the load balancer URL to Solr.
>>
>> 1. Get cluster status.
>> 2. Parse out the Zookeeper config string including chroot.
>> 3. Connect to Zookeeper.
>> 4. Copy the config to the location described in Shawn’s message.
>> 5. Send linkconfig command to the cluster, just to be sure.
>> 6. Reload the collection with an async command.
>> 7. Ping the cluster until the reload is successful on every node.
>> 8. Optionally, rebuild the suggester on each node.
>>
>> The actual location of the config in Zookeeper is undocumented, as far as
>> I could tell. I used the Solr ZK CLI, then reverse engineered where it put
>> stuff.
>>
>> The docs need a “Zookeeper file organization” chapter with this info.
>>
>> Also, it would be nice if the ZKHOST info was available pre-parsed in
>> cluster status.
>>
>> wunder
>> Walter Underwood
>> [hidden email]
>> http://observer.wunderwood.org/  (my blog)
>>
>> > On Apr 17, 2018, at 8:20 PM, Shawn Heisey <[hidden email]> wrote:
>> >
>> > On 4/17/2018 8:54 PM, Aristedes Maniatis wrote:
>> >> Is there any difference between using the tools supplied with Solr to
>> write configuration to Zookeeper or just writing directly to our Zookeeper
>> cluster?
>> >>
>> >> We have tooling that makes it much easier to write directly to ZK
>> rather than having to use yet another tool to do it.
>> >
>> > As long as it ends up in the correct path in the ZK structure, it
>> doesn't matter how it gets there.
>> >
>> > The /configs/XXXX location (where XXXX is the config name) should have
>> the same contents that would normally be found in a conf directory if it
>> were standalone Solr and not using the standalone configsets feature.
>> >
>> > Thanks,
>> > Shawn
>> >
>>
>>
Reply | Threaded
Open this post in threaded view
|

Re: Writing config directly to zookeeper

Walter Underwood
I didn’t want to install Solr just so Jenkins could use one script. The Python is standalone.

I was using the zkCli tools, which were just not all that well documented. I never could find a description of exactly which files were copied where. The solr.xml directory structure had <core>/conf/<files>, but it wasn’t clear what was expected for the bootstrap commands.

Fetching the zk information from a running cluster is also less error prone. Don’t need to keep Jenkins configured the same as the cluster.

Oh, I skipped a step, the Python script also uploads solr.xml.

wunder
Walter Underwood
[hidden email]
http://observer.wunderwood.org/  (my blog)

> On Apr 18, 2018, at 9:14 AM, Erick Erickson <[hidden email]> wrote:
>
> There are some perhaps easier ways to manipulate ZK in the "bin/solr"
> script if you haven't seen it
>
> bin/solr zk -help
>
> Best,
> Erick
>
> On Wed, Apr 18, 2018 at 8:30 AM, Arturas Mazeika <[hidden email]> wrote:
>> Hi Walter,
>>
>> Thanks for the message. Would you care to share the tool with us? I would
>> be interested.. Or have you shared it already?
>>
>> Cheers,
>> Arturas
>>
>> On Wed, Apr 18, 2018 at 5:09 PM, Walter Underwood <[hidden email]>
>> wrote:
>>
>>> I wrote a Python tool to do this. I use the kazoo package to talk to
>>> Zookeeper. It starts with the load balancer URL to Solr.
>>>
>>> 1. Get cluster status.
>>> 2. Parse out the Zookeeper config string including chroot.
>>> 3. Connect to Zookeeper.
>>> 4. Copy the config to the location described in Shawn’s message.
>>> 5. Send linkconfig command to the cluster, just to be sure.
>>> 6. Reload the collection with an async command.
>>> 7. Ping the cluster until the reload is successful on every node.
>>> 8. Optionally, rebuild the suggester on each node.
>>>
>>> The actual location of the config in Zookeeper is undocumented, as far as
>>> I could tell. I used the Solr ZK CLI, then reverse engineered where it put
>>> stuff.
>>>
>>> The docs need a “Zookeeper file organization” chapter with this info.
>>>
>>> Also, it would be nice if the ZKHOST info was available pre-parsed in
>>> cluster status.
>>>
>>> wunder
>>> Walter Underwood
>>> [hidden email]
>>> http://observer.wunderwood.org/  (my blog)
>>>
>>>> On Apr 17, 2018, at 8:20 PM, Shawn Heisey <[hidden email]> wrote:
>>>>
>>>> On 4/17/2018 8:54 PM, Aristedes Maniatis wrote:
>>>>> Is there any difference between using the tools supplied with Solr to
>>> write configuration to Zookeeper or just writing directly to our Zookeeper
>>> cluster?
>>>>>
>>>>> We have tooling that makes it much easier to write directly to ZK
>>> rather than having to use yet another tool to do it.
>>>>
>>>> As long as it ends up in the correct path in the ZK structure, it
>>> doesn't matter how it gets there.
>>>>
>>>> The /configs/XXXX location (where XXXX is the config name) should have
>>> the same contents that would normally be found in a conf directory if it
>>> were standalone Solr and not using the standalone configsets feature.
>>>>
>>>> Thanks,
>>>> Shawn
>>>>
>>>
>>>