Programmatically find out if node is overseer

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

Programmatically find out if node is overseer

Markus Jelsma-2
Hello - i need to run a thread on a single instance of a cloud so need to find out if current node is the overseer. I know we can already programmatically find out if this replica is the leader of a shard via isLeader(). I have looked everywhere but i cannot find an isOverseer. I did find the election stuff but i am unsure if that is what i need to use.

Any thoughts?

Thanks!
Markus
Reply | Threaded
Open this post in threaded view
|

Re: Programmatically find out if node is overseer

Erick Erickson
look at the overseer election ephemeral node in ZK, the first one in
line is the current overseer.

Best,
Erick

On Thu, Jul 16, 2015 at 3:42 AM, Markus Jelsma
<[hidden email]> wrote:
> Hello - i need to run a thread on a single instance of a cloud so need to find out if current node is the overseer. I know we can already programmatically find out if this replica is the leader of a shard via isLeader(). I have looked everywhere but i cannot find an isOverseer. I did find the election stuff but i am unsure if that is what i need to use.
>
> Any thoughts?
>
> Thanks!
> Markus
Reply | Threaded
Open this post in threaded view
|

Re: Programmatically find out if node is overseer

Shai Erera
An easier way (IMO) and more 'official' is to use the CLUSTERSTATUS (
https://cwiki.apache.org/confluence/display/solr/Collections+API#CollectionsAPI-api18)
or OVERSEERSTATUS (
https://cwiki.apache.org/confluence/display/solr/Collections+API#CollectionsAPI-api17)
API.

The OVERSEERSTATUS returns a 'leader' item which says who is the overseer,
at least as far as I understand. Not sure what is returned in case there
are multiple nodes with the overseer role.

The CLUSTERSTATUS returns an 'overseer' item with all nodes that have the
overseer role assigned. I'm usually using that API to query for the status
of my Solr cluster.

Shai

On Fri, Jul 17, 2015 at 3:55 AM, Erick Erickson <[hidden email]>
wrote:

> look at the overseer election ephemeral node in ZK, the first one in
> line is the current overseer.
>
> Best,
> Erick
>
> On Thu, Jul 16, 2015 at 3:42 AM, Markus Jelsma
> <[hidden email]> wrote:
> > Hello - i need to run a thread on a single instance of a cloud so need
> to find out if current node is the overseer. I know we can already
> programmatically find out if this replica is the leader of a shard via
> isLeader(). I have looked everywhere but i cannot find an isOverseer. I did
> find the election stuff but i am unsure if that is what i need to use.
> >
> > Any thoughts?
> >
> > Thanks!
> > Markus
>
Reply | Threaded
Open this post in threaded view
|

Re: Programmatically find out if node is overseer

Erick Erickson
good point Shai!

On Thu, Jul 16, 2015 at 9:36 PM, Shai Erera <[hidden email]> wrote:

> An easier way (IMO) and more 'official' is to use the CLUSTERSTATUS (
> https://cwiki.apache.org/confluence/display/solr/Collections+API#CollectionsAPI-api18)
> or OVERSEERSTATUS (
> https://cwiki.apache.org/confluence/display/solr/Collections+API#CollectionsAPI-api17)
> API.
>
> The OVERSEERSTATUS returns a 'leader' item which says who is the overseer,
> at least as far as I understand. Not sure what is returned in case there
> are multiple nodes with the overseer role.
>
> The CLUSTERSTATUS returns an 'overseer' item with all nodes that have the
> overseer role assigned. I'm usually using that API to query for the status
> of my Solr cluster.
>
> Shai
>
> On Fri, Jul 17, 2015 at 3:55 AM, Erick Erickson <[hidden email]>
> wrote:
>
>> look at the overseer election ephemeral node in ZK, the first one in
>> line is the current overseer.
>>
>> Best,
>> Erick
>>
>> On Thu, Jul 16, 2015 at 3:42 AM, Markus Jelsma
>> <[hidden email]> wrote:
>> > Hello - i need to run a thread on a single instance of a cloud so need
>> to find out if current node is the overseer. I know we can already
>> programmatically find out if this replica is the leader of a shard via
>> isLeader(). I have looked everywhere but i cannot find an isOverseer. I did
>> find the election stuff but i am unsure if that is what i need to use.
>> >
>> > Any thoughts?
>> >
>> > Thanks!
>> > Markus
>>
Reply | Threaded
Open this post in threaded view
|

Re: Programmatically find out if node is overseer

Anshum Gupta
In reply to this post by Shai Erera
As Shai mentioned, OVERSEERSTATUS is the most straight forward and
recommended way to go. It basically does what Erick suggested i.e. get the
first entry from '/overseer_elect/leader' in zk.

Also, ideally, there shouldn't be a point where you have multiple active
Overseers in a single cluster.

On Thu, Jul 16, 2015 at 9:36 PM, Shai Erera <[hidden email]> wrote:

> An easier way (IMO) and more 'official' is to use the CLUSTERSTATUS (
>
> https://cwiki.apache.org/confluence/display/solr/Collections+API#CollectionsAPI-api18
> )
> or OVERSEERSTATUS (
>
> https://cwiki.apache.org/confluence/display/solr/Collections+API#CollectionsAPI-api17
> )
> API.
>
> The OVERSEERSTATUS returns a 'leader' item which says who is the overseer,
> at least as far as I understand. Not sure what is returned in case there
> are multiple nodes with the overseer role.
>
> The CLUSTERSTATUS returns an 'overseer' item with all nodes that have the
> overseer role assigned. I'm usually using that API to query for the status
> of my Solr cluster.
>
> Shai
>
> On Fri, Jul 17, 2015 at 3:55 AM, Erick Erickson <[hidden email]>
> wrote:
>
> > look at the overseer election ephemeral node in ZK, the first one in
> > line is the current overseer.
> >
> > Best,
> > Erick
> >
> > On Thu, Jul 16, 2015 at 3:42 AM, Markus Jelsma
> > <[hidden email]> wrote:
> > > Hello - i need to run a thread on a single instance of a cloud so need
> > to find out if current node is the overseer. I know we can already
> > programmatically find out if this replica is the leader of a shard via
> > isLeader(). I have looked everywhere but i cannot find an isOverseer. I
> did
> > find the election stuff but i am unsure if that is what i need to use.
> > >
> > > Any thoughts?
> > >
> > > Thanks!
> > > Markus
> >
>



--
Anshum Gupta
Reply | Threaded
Open this post in threaded view
|

Re: Programmatically find out if node is overseer

SolrUser2015
Hi Anshum what do you mean by:
>ideally, there shouldn't be a point where you have multiple active
Overseers in a single cluster

How can multiple Overseers happen? And what are the consequences?

Regards

> On 17 Jul 2015, at 19:37, Anshum Gupta <[hidden email]> wrote:
>
> ideally, there shouldn't be a point where you have multiple active
> Overseers in a single cluster
Reply | Threaded
Open this post in threaded view
|

Re: Programmatically find out if node is overseer

Anshum Gupta
It shouldn't happen unless you're using an older version of Solr (< 4.8) in
which case, you might end up hitting SOLR-5859
<https://issues.apache.org/jira/browse/SOLR-5859>.

On Fri, Jul 17, 2015 at 11:29 AM, <[hidden email]> wrote:

> Hi Anshum what do you mean by:
> >ideally, there shouldn't be a point where you have multiple active
> Overseers in a single cluster
>
> How can multiple Overseers happen? And what are the consequences?
>
> Regards
>
> > On 17 Jul 2015, at 19:37, Anshum Gupta <[hidden email]> wrote:
> >
> > ideally, there shouldn't be a point where you have multiple active
> > Overseers in a single cluster
>



--
Anshum Gupta
Reply | Threaded
Open this post in threaded view
|

Re: Programmatically find out if node is overseer

Shalin Shekhar Mangar
Just for the record, SOLR-5859 only affects people who used the
overseer roles feature released in 4.7 and no one else. This was fixed
in 4.8

On Sat, Jul 18, 2015 at 12:18 AM, Anshum Gupta <[hidden email]> wrote:

> It shouldn't happen unless you're using an older version of Solr (< 4.8) in
> which case, you might end up hitting SOLR-5859
> <https://issues.apache.org/jira/browse/SOLR-5859>.
>
> On Fri, Jul 17, 2015 at 11:29 AM, <[hidden email]> wrote:
>
>> Hi Anshum what do you mean by:
>> >ideally, there shouldn't be a point where you have multiple active
>> Overseers in a single cluster
>>
>> How can multiple Overseers happen? And what are the consequences?
>>
>> Regards
>>
>> > On 17 Jul 2015, at 19:37, Anshum Gupta <[hidden email]> wrote:
>> >
>> > ideally, there shouldn't be a point where you have multiple active
>> > Overseers in a single cluster
>>
>
>
>
> --
> Anshum Gupta



--
Regards,
Shalin Shekhar Mangar.
Reply | Threaded
Open this post in threaded view
|

Re: Programmatically find out if node is overseer

Shai Erera
>
> Also, ideally, there shouldn't be a point where you have multiple active
> Overseers in a single cluster.
>

In the reference guide, CLUSTERSTATUS shows as if the overseer role can
return more than one node. Does it mean that these nodes were designated
potential 'overseers', but OVERSEERSTATUS' will return the actual one?

Shai

On Fri, Jul 17, 2015 at 10:11 PM, Shalin Shekhar Mangar <
[hidden email]> wrote:

> Just for the record, SOLR-5859 only affects people who used the
> overseer roles feature released in 4.7 and no one else. This was fixed
> in 4.8
>
> On Sat, Jul 18, 2015 at 12:18 AM, Anshum Gupta <[hidden email]>
> wrote:
> > It shouldn't happen unless you're using an older version of Solr (< 4.8)
> in
> > which case, you might end up hitting SOLR-5859
> > <https://issues.apache.org/jira/browse/SOLR-5859>.
> >
> > On Fri, Jul 17, 2015 at 11:29 AM, <[hidden email]> wrote:
> >
> >> Hi Anshum what do you mean by:
> >> >ideally, there shouldn't be a point where you have multiple active
> >> Overseers in a single cluster
> >>
> >> How can multiple Overseers happen? And what are the consequences?
> >>
> >> Regards
> >>
> >> > On 17 Jul 2015, at 19:37, Anshum Gupta <[hidden email]>
> wrote:
> >> >
> >> > ideally, there shouldn't be a point where you have multiple active
> >> > Overseers in a single cluster
> >>
> >
> >
> >
> > --
> > Anshum Gupta
>
>
>
> --
> Regards,
> Shalin Shekhar Mangar.
>
Reply | Threaded
Open this post in threaded view
|

Re: Programmatically find out if node is overseer

Shalin Shekhar Mangar
On Sat, Jul 18, 2015 at 1:00 AM, Shai Erera <[hidden email]> wrote:
>>
>> Also, ideally, there shouldn't be a point where you have multiple active
>> Overseers in a single cluster.
>>
>
> In the reference guide, CLUSTERSTATUS shows as if the overseer role can
> return more than one node. Does it mean that these nodes were designated
> potential 'overseers', but OVERSEERSTATUS' will return the actual one?

Yes, the OVERSEERSTATUS will return the current (actual) leader always.

>
> Shai
>
> On Fri, Jul 17, 2015 at 10:11 PM, Shalin Shekhar Mangar <
> [hidden email]> wrote:
>
>> Just for the record, SOLR-5859 only affects people who used the
>> overseer roles feature released in 4.7 and no one else. This was fixed
>> in 4.8
>>
>> On Sat, Jul 18, 2015 at 12:18 AM, Anshum Gupta <[hidden email]>
>> wrote:
>> > It shouldn't happen unless you're using an older version of Solr (< 4.8)
>> in
>> > which case, you might end up hitting SOLR-5859
>> > <https://issues.apache.org/jira/browse/SOLR-5859>.
>> >
>> > On Fri, Jul 17, 2015 at 11:29 AM, <[hidden email]> wrote:
>> >
>> >> Hi Anshum what do you mean by:
>> >> >ideally, there shouldn't be a point where you have multiple active
>> >> Overseers in a single cluster
>> >>
>> >> How can multiple Overseers happen? And what are the consequences?
>> >>
>> >> Regards
>> >>
>> >> > On 17 Jul 2015, at 19:37, Anshum Gupta <[hidden email]>
>> wrote:
>> >> >
>> >> > ideally, there shouldn't be a point where you have multiple active
>> >> > Overseers in a single cluster
>> >>
>> >
>> >
>> >
>> > --
>> > Anshum Gupta
>>
>>
>>
>> --
>> Regards,
>> Shalin Shekhar Mangar.
>>



--
Regards,
Shalin Shekhar Mangar.
Reply | Threaded
Open this post in threaded view
|

Re: Programmatically find out if node is overseer

Vincenzo D'Amore
Hi, in solrj I did this to find leaders url :

CloudSolrServer solr = (CloudSolrServer) Configurazione.getSolrServer();
String collection = "collection1";

Map<String,String> collectionAliases =
solr.getZkStateReader().getAliases().getCollectionAliasMap();
if (collectionAliases.containsKey(collection)) {
corename = collectionAliases.get(collection);
 for (Slice slice :
solr.getZkStateReader().getClusterState().getSlices(corename)) {
Replica sliceLeader = slice.getLeader();
log.info("faccio backup su {} {}", sliceLeader.getNodeName(),
sliceLeader.getProperties().get("core"));
                        log.info(String.format("%s/%s/",sliceLeader.get(
"base_url"), sliceLeader.getProperties().get("core")));
}
        }


On Fri, Jul 17, 2015 at 9:37 PM, Shalin Shekhar Mangar <
[hidden email]> wrote:

> On Sat, Jul 18, 2015 at 1:00 AM, Shai Erera <[hidden email]> wrote:
> >>
> >> Also, ideally, there shouldn't be a point where you have multiple active
> >> Overseers in a single cluster.
> >>
> >
> > In the reference guide, CLUSTERSTATUS shows as if the overseer role can
> > return more than one node. Does it mean that these nodes were designated
> > potential 'overseers', but OVERSEERSTATUS' will return the actual one?
>
> Yes, the OVERSEERSTATUS will return the current (actual) leader always.
>
> >
> > Shai
> >
> > On Fri, Jul 17, 2015 at 10:11 PM, Shalin Shekhar Mangar <
> > [hidden email]> wrote:
> >
> >> Just for the record, SOLR-5859 only affects people who used the
> >> overseer roles feature released in 4.7 and no one else. This was fixed
> >> in 4.8
> >>
> >> On Sat, Jul 18, 2015 at 12:18 AM, Anshum Gupta <[hidden email]>
> >> wrote:
> >> > It shouldn't happen unless you're using an older version of Solr (<
> 4.8)
> >> in
> >> > which case, you might end up hitting SOLR-5859
> >> > <https://issues.apache.org/jira/browse/SOLR-5859>.
> >> >
> >> > On Fri, Jul 17, 2015 at 11:29 AM, <[hidden email]> wrote:
> >> >
> >> >> Hi Anshum what do you mean by:
> >> >> >ideally, there shouldn't be a point where you have multiple active
> >> >> Overseers in a single cluster
> >> >>
> >> >> How can multiple Overseers happen? And what are the consequences?
> >> >>
> >> >> Regards
> >> >>
> >> >> > On 17 Jul 2015, at 19:37, Anshum Gupta <[hidden email]>
> >> wrote:
> >> >> >
> >> >> > ideally, there shouldn't be a point where you have multiple active
> >> >> > Overseers in a single cluster
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > Anshum Gupta
> >>
> >>
> >>
> >> --
> >> Regards,
> >> Shalin Shekhar Mangar.
> >>
>
>
>
> --
> Regards,
> Shalin Shekhar Mangar.
>



--
Vincenzo D'Amore
email: [hidden email]
skype: free.dev
mobile: +39 349 8513251
Reply | Threaded
Open this post in threaded view
|

Re: Programmatically find out if node is overseer

Chris Hostetter-3
In reply to this post by Markus Jelsma-2

: Hello - i need to run a thread on a single instance of a cloud so need
: to find out if current node is the overseer. I know we can already
: programmatically find out if this replica is the leader of a shard via
: isLeader(). I have looked everywhere but i cannot find an isOverseer. I

At one point, i woked up a utility method to give internal plugins
access to an "isOverseer()" type utility method...

   https://issues.apache.org/jira/browse/SOLR-5823

...but ultimately i abandoned this because i was completley forgetting
(until much much too late) that there's really no reason to assume that
any/all collections will have a single shard on the same node as the
overseer -- so having a plugin that only does stuff if it's running on the
overseer node is a really bad idea, because it might not run at all. (even
if it's configured in every collection)


what i ultimately wound up doing (see SOLR-5795) is implementing a
solution where every core (of each collection configured to want this
functionality) has a thread running (a TimedExecutor) which would do
nothing unless...
 * my slice is active? (ie: not in the process of being shut down)
 * my slice is 'first' in a sorted list of slices?
 * i am currently the leader of my slice?

...that way when the timer goes off ever X minutes, at *most* one thread
fires (we might sporadically get no evens triggered if/when there is
leader election in progress for the slice that matters)

the choice of "first" slice name alphabetically is purely becuase it's
something cheap to compute and garunteeded to be unique.


If you truly want exactly one thread for the entire cluster, regardless of
collection, you could do the same basic idea by just adding a "my
collection is 'first' in a sorted list of collection names?"



-Hoss
http://www.lucidworks.com/
Reply | Threaded
Open this post in threaded view
|

RE: Programmatically find out if node is overseer

Markus Jelsma-2
In reply to this post by Markus Jelsma-2
Hello - this approach not only solves the problem but also allows me to run different processing threads on other nodes.

Thanks!
Markus
 
-----Original message-----

> From:Chris Hostetter <[hidden email]>
> Sent: Saturday 18th July 2015 1:00
> To: solr-user <[hidden email]>
> Subject: Re: Programmatically find out if node is overseer
>
>
> : Hello - i need to run a thread on a single instance of a cloud so need
> : to find out if current node is the overseer. I know we can already
> : programmatically find out if this replica is the leader of a shard via
> : isLeader(). I have looked everywhere but i cannot find an isOverseer. I
>
> At one point, i woked up a utility method to give internal plugins
> access to an "isOverseer()" type utility method...
>
>    https://issues.apache.org/jira/browse/SOLR-5823
>
> ...but ultimately i abandoned this because i was completley forgetting
> (until much much too late) that there's really no reason to assume that
> any/all collections will have a single shard on the same node as the
> overseer -- so having a plugin that only does stuff if it's running on the
> overseer node is a really bad idea, because it might not run at all. (even
> if it's configured in every collection)
>
>
> what i ultimately wound up doing (see SOLR-5795) is implementing a
> solution where every core (of each collection configured to want this
> functionality) has a thread running (a TimedExecutor) which would do
> nothing unless...
>  * my slice is active? (ie: not in the process of being shut down)
>  * my slice is 'first' in a sorted list of slices?
>  * i am currently the leader of my slice?
>
> ...that way when the timer goes off ever X minutes, at *most* one thread
> fires (we might sporadically get no evens triggered if/when there is
> leader election in progress for the slice that matters)
>
> the choice of "first" slice name alphabetically is purely becuase it's
> something cheap to compute and garunteeded to be unique.
>
>
> If you truly want exactly one thread for the entire cluster, regardless of
> collection, you could do the same basic idea by just adding a "my
> collection is 'first' in a sorted list of collection names?"
>
>
>
> -Hoss
> http://www.lucidworks.com/
>