HA and load balancing Qauestion

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

HA and load balancing Qauestion

Venkatesh Seetharam
Hello there,

Howdy. I'd like to know if I can configure Multiple Solr instances working
with a single read-only index partition for failover/HA and load balancing
purposes. Or is there any other way that Solr has built-in features to
handle the same.

Any ideas/thoughts are greatly appreciated.

--
Thanks,
Venkatesh

"Perfection (in design) is achieved not when there is nothing more to add,
but rather when there is nothing more to take away."
- Antoine de Saint-Exupéry
Reply | Threaded
Open this post in threaded view
|

Re: HA and load balancing Qauestion

Yonik Seeley-2
On 3/8/07, Venkatesh Seetharam <[hidden email]> wrote:
> Howdy. I'd like to know if I can configure Multiple Solr instances working
> with a single read-only index partition for failover/HA and load balancing
> purposes. Or is there any other way that Solr has built-in features to
> handle the same.

On the front-end, HTTP is easily load-balanced via software or
hardware loadbalancers.

To distribute a single index to multiple solr searchers, see
http://wiki.apache.org/solr/CollectionDistribution

You don't have to do it that way though... if you have another
mechanism to get the index to the searchers, that could work too.

-Yonik
Reply | Threaded
Open this post in threaded view
|

Re: HA and load balancing Qauestion

Venkatesh Seetharam
Thanks for the reply Yonik.

I'm not using HTTP and using a wrapper to wrap Solr for searching. I'm using
RPC to talk to multiple servers.

Can I point 2 Solr instances to the same index partition, having the same
path in SolrConfig? Is this safe or I need to make 2 copies of the same
index partition and point the Solr instances to these copies? Since my index
partition lives on a shared NetApp mount, I'd like to use the same index
partition for multiple Solr instances.

Thanks for any help,
Venkatesh

On 3/8/07, Yonik Seeley <[hidden email]> wrote:

>
> On 3/8/07, Venkatesh Seetharam <[hidden email]> wrote:
> > Howdy. I'd like to know if I can configure Multiple Solr instances
> working
> > with a single read-only index partition for failover/HA and load
> balancing
> > purposes. Or is there any other way that Solr has built-in features to
> > handle the same.
>
> On the front-end, HTTP is easily load-balanced via software or
> hardware loadbalancers.
>
> To distribute a single index to multiple solr searchers, see
> http://wiki.apache.org/solr/CollectionDistribution
>
> You don't have to do it that way though... if you have another
> mechanism to get the index to the searchers, that could work too.
>
> -Yonik
>
Reply | Threaded
Open this post in threaded view
|

Re: HA and load balancing Qauestion

Yonik Seeley-2
On 3/8/07, Venkatesh Seetharam <[hidden email]> wrote:
> Can I point 2 Solr instances to the same index partition, having the same
> path in SolrConfig?

Yes, that should work fine.

-Yonik
Reply | Threaded
Open this post in threaded view
|

Re: HA and load balancing Qauestion

Chris Hostetter-3

: > Can I point 2 Solr instances to the same index partition, having the same
: > path in SolrConfig?
:
: Yes, that should work fine.

you might run into some weirdness if you send updates/delets to both
instances .. basically you'll want to configure all but one instace as a
"slave" and anytime you do a commit on the master you'll want to trigger
a commit on all of the slaves so that they reopen the index.

(just like using the snapshot scripts, except you don't need to snapshoot,
snappull, or snapinstall)



-Hoss

Reply | Threaded
Open this post in threaded view
|

Re: HA and load balancing Qauestion

Venkatesh Seetharam
Thanks Yonik and Chris for your confirmation. Chris, these are read-only
index partitions. I perform updates/deletions on a master index which will
be snapshotted at some fixed intervals. I'll look into the Collection
Distribution of Solr. Sounds very powerful.

I'm struck with Solr requiring an index directory under dataDir configured
in SolrConfig.  Why does it not take a complete path_to_index configured
under dataDir but append "index"? Is there anyway I can workaround this?

org.apache.solr.core.SolrCore:  this.index_path = dataDir + "/" + "index";

Thanks,
Venkatesh

On 3/8/07, Chris Hostetter <[hidden email]> wrote:

>
>
> : > Can I point 2 Solr instances to the same index partition, having the
> same
> : > path in SolrConfig?
> :
> : Yes, that should work fine.
>
> you might run into some weirdness if you send updates/delets to both
> instances .. basically you'll want to configure all but one instace as a
> "slave" and anytime you do a commit on the master you'll want to trigger
> a commit on all of the slaves so that they reopen the index.
>
> (just like using the snapshot scripts, except you don't need to snapshoot,
> snappull, or snapinstall)
>
>
>
> -Hoss
>
>
Reply | Threaded
Open this post in threaded view
|

Re: HA and load balancing Qauestion

Chris Hostetter-3

: I'm struck with Solr requiring an index directory under dataDir configured
: in SolrConfig.  Why does it not take a complete path_to_index configured
: under dataDir but append "index"? Is there anyway I can workaround this?

i think at one time we were assuming there might be other types of data
you'd want Solr to store besides the index... the assumption is that you
tell Solr where you want it to keep all of it's data, and after that you
shouldn't care what lives in that directory.

if i remember correctly, the dataDir is alwo where the solr snapshots and
temp dirs get put ... if Solr let you configure the indexdir directly,
we'd need you to also configure those locations seperately -- except that
they have to be on the same physical disk for hardlinks to work, so it's
relaly just a lot simpler if you tell SOlr the dataDir and let it take
care of everything else.

(except in your case where you *want* to take care of it ... Solr wasn't
really designed for that case)



-Hoss

Reply | Threaded
Open this post in threaded view
|

Re: HA and load balancing Qauestion

Venkatesh Seetharam
Thanks Hoss for the clarification. I think I can make a copy of the index
for searching and rename 'em. I think I can work around this one but good to
know the bigger picture.

Venkatesh

On 3/8/07, Chris Hostetter <[hidden email]> wrote:

>
>
> : I'm struck with Solr requiring an index directory under dataDir
> configured
> : in SolrConfig.  Why does it not take a complete path_to_index configured
> : under dataDir but append "index"? Is there anyway I can workaround this?
>
> i think at one time we were assuming there might be other types of data
> you'd want Solr to store besides the index... the assumption is that you
> tell Solr where you want it to keep all of it's data, and after that you
> shouldn't care what lives in that directory.
>
> if i remember correctly, the dataDir is alwo where the solr snapshots and
> temp dirs get put ... if Solr let you configure the indexdir directly,
> we'd need you to also configure those locations seperately -- except that
> they have to be on the same physical disk for hardlinks to work, so it's
> relaly just a lot simpler if you tell SOlr the dataDir and let it take
> care of everything else.
>
> (except in your case where you *want* to take care of it ... Solr wasn't
> really designed for that case)
>
>
>
> -Hoss
>
>