Slave/Master swap

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Slave/Master swap

Otis Gospodnetic-2
Hi,



I saw https://issues.apache.org/jira/browse/SOLR-265 (Make IndexSchema updateable in live system) which made me think of something I wished for a while back.

Having a single Solr Master and a couple of Solr Slaves is a common setup.  If any of the Slaves fails, a decent LB knows not to talk to it until it's back up.  What happens when the single Solr Master fails?  One (cheap) way to deal with that might be to promote one of the Solr Slaves to the new Master role.  If the snapshooter script is called manually on the Master, the appropriate monitoring tools would need to start the same calls on the new Master (former Slave) box.  But if the snapshooter is configured via solrconfig.xml to run after commit and/or optimize, we'd have to swap solrconfig.xml and restart Solr on the ex-Slave to make it the new Master (and also make some changes in the LB VIPs, most likely).

I'm wondering if there are slicker ways to do this, ways that would minimize the downtime, for instance.  Perhaps, just like Will Johnson is trying to make IndexSchema updateable in a live system, the snapshooter could be turned on/off programatically, say via a special request handler.

Thanks,
Otis



Reply | Threaded
Open this post in threaded view
|

Re: Slave/Master swap

Chris Hostetter-3

: I'm wondering if there are slicker ways to do this, ways that would
: minimize the downtime, for instance.  Perhaps, just like Will Johnson is
: trying to make IndexSchema updateable in a live system, the snapshooter
: could be turned on/off programatically, say via a special request
: handler.

and easy way to do that would be to modify the configuration of
RunExecutableListener in the solrconfig.xml to execute a wrapper script
around snapshooter that only runs it if a flag file exists on disk.

the problem is there are other things you typically want differnet between
a master and a slave ... uses of the QuerySenderListener (it could also be
modified to check for flag file i suppose)m cache sizes, and cache
autowarming.



-Hoss

Reply | Threaded
Open this post in threaded view
|

Re: Slave/Master swap

Otis Gospodnetic-2
In reply to this post by Otis Gospodnetic-2
Hi,

Yes, I thought of flag file + wrapper script tricks, but that didn't sound super elegant either, and the other differences in behaviour between master and slave are also true.

Hmmmm, I've always wanted to try DRDB (http://www.drbd.org/).  Master->(Master+Slaves) replication via DRDB?  I imagine it would be expensive...

So if I want to turn a Slave into a Master, the best thing to do is to swap solrconfigs and restart the ex-Slave to turn it into a Master.  The more expensive solution might be to have Solr instances run on top of a SAN and then one could really have multiple Master instances, one in stand-by mode and ready to be started as the new Master if the current Master decides to go on vacation.  Any flaws there?  Out of curiosity, how does CNet handle Master redundancy?

Otis


----- Original Message ----
From: Chris Hostetter <[hidden email]>
To: [hidden email]
Sent: Wednesday, June 20, 2007 9:40:51 PM
Subject: Re: Slave/Master swap


: I'm wondering if there are slicker ways to do this, ways that would
: minimize the downtime, for instance.  Perhaps, just like Will Johnson is
: trying to make IndexSchema updateable in a live system, the snapshooter
: could be turned on/off programatically, say via a special request
: handler.

and easy way to do that would be to modify the configuration of
RunExecutableListener in the solrconfig.xml to execute a wrapper script
around snapshooter that only runs it if a flag file exists on disk.

the problem is there are other things you typically want differnet between
a master and a slave ... uses of the QuerySenderListener (it could also be
modified to check for flag file i suppose)m cache sizes, and cache
autowarming.



-Hoss




Reply | Threaded
Open this post in threaded view
|

Re: Slave/Master swap

Chris Hostetter-3


: The more expensive solution might be to have Solr instances run on top
: of a SAN and then one could really have multiple Master instances, one
: in stand-by mode and ready to be started as the new Master if the

i *believe* that if you have two solr isntances pointed at the same
physical data directory (SAN or otherwise) but you only send update/commit
commands to one, they won't interfere with eachother.  so concievable you
can have both masters up and running and your "failover" approach if the
primary goes down is just to start sending updates to the secondary.
you'll loose any unflushed changes that hte primary had in memory, but
those are lost anyway.

don't trust me on that though, test it out yourself.

: curiosity, how does CNet handle Master redundancy?

I don't know how much i'm allowed to talk about our processes and systems
for redundency, disastery recovery, fallover, etc... but i don't think
i'll upset anyone if i tell you: as far as i know, we've never needed to
take advantage of them with a solr master.  ie: we've never had a solr
master crash so hard we had to bring up another one in it's place ...
knock on wood.  (that probably has more to do with having good hardware
then anything else though).

(and no, i honestly don't know what hardware we use ... i don't bother
paying attention, i let hte hardware guys worry about that)


-Hoss

Reply | Threaded
Open this post in threaded view
|

Re: Slave/Master swap

Otis Gospodnetic-2
In reply to this post by Otis Gospodnetic-2
Right, that SAN con 2 Masters sounds good.  Lucky you with your lonely Master!  Where I work hw failures are pretty common.

Otis
 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Simpy -- http://www.simpy.com/  -  Tag  -  Search  -  Share

----- Original Message ----
From: Chris Hostetter <[hidden email]>
To: [hidden email]
Sent: Wednesday, June 20, 2007 11:43:02 PM
Subject: Re: Slave/Master swap



: The more expensive solution might be to have Solr instances run on top
: of a SAN and then one could really have multiple Master instances, one
: in stand-by mode and ready to be started as the new Master if the

i *believe* that if you have two solr isntances pointed at the same
physical data directory (SAN or otherwise) but you only send update/commit
commands to one, they won't interfere with eachother.  so concievable you
can have both masters up and running and your "failover" approach if the
primary goes down is just to start sending updates to the secondary.
you'll loose any unflushed changes that hte primary had in memory, but
those are lost anyway.

don't trust me on that though, test it out yourself.

: curiosity, how does CNet handle Master redundancy?

I don't know how much i'm allowed to talk about our processes and systems
for redundency, disastery recovery, fallover, etc... but i don't think
i'll upset anyone if i tell you: as far as i know, we've never needed to
take advantage of them with a solr master.  ie: we've never had a solr
master crash so hard we had to bring up another one in it's place ...
knock on wood.  (that probably has more to do with having good hardware
then anything else though).

(and no, i honestly don't know what hardware we use ... i don't bother
paying attention, i let hte hardware guys worry about that)


-Hoss




Reply | Threaded
Open this post in threaded view
|

Re: Slave/Master swap

James liu-2
If just one master or  one slave server fail, i think u maybe can use master
index server.

shell controlled by program is easy for me. i use php  and shell_exec.


2007/6/21, Otis Gospodnetic <[hidden email]>:

>
> Right, that SAN con 2 Masters sounds good.  Lucky you with your lonely
> Master!  Where I work hw failures are pretty common.
>
> Otis
> . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
> Simpy -- http://www.simpy.com/  -  Tag  -  Search  -  Share
>
> ----- Original Message ----
> From: Chris Hostetter <[hidden email]>
> To: [hidden email]
> Sent: Wednesday, June 20, 2007 11:43:02 PM
> Subject: Re: Slave/Master swap
>
>
>
> : The more expensive solution might be to have Solr instances run on top
> : of a SAN and then one could really have multiple Master instances, one
> : in stand-by mode and ready to be started as the new Master if the
>
> i *believe* that if you have two solr isntances pointed at the same
> physical data directory (SAN or otherwise) but you only send update/commit
> commands to one, they won't interfere with eachother.  so concievable you
> can have both masters up and running and your "failover" approach if the
> primary goes down is just to start sending updates to the secondary.
> you'll loose any unflushed changes that hte primary had in memory, but
> those are lost anyway.
>
> don't trust me on that though, test it out yourself.
>
> : curiosity, how does CNet handle Master redundancy?
>
> I don't know how much i'm allowed to talk about our processes and systems
> for redundency, disastery recovery, fallover, etc... but i don't think
> i'll upset anyone if i tell you: as far as i know, we've never needed to
> take advantage of them with a solr master.  ie: we've never had a solr
> master crash so hard we had to bring up another one in it's place ...
> knock on wood.  (that probably has more to do with having good hardware
> then anything else though).
>
> (and no, i honestly don't know what hardware we use ... i don't bother
> paying attention, i let hte hardware guys worry about that)
>
>
> -Hoss
>
>
>
>
>


--
regards
jl