Shared index base

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

Shared index base

Evgeniy Strokin
I know there was such discussions about the subject, but I want to ask again if somebody could share more information.
We are planning to have several separate servers for our search engine. One of them will be index/search server, and all others are search only.
We want to use SAN (BTW: should we consider something else?) and give access to it from all servers. So all servers will use the same index base, without any replication, same files.
Is this a good practice? Did somebody do the same? Any problems noticed? Or any suggestions, even about different configurations are highly appreciated.
 
Thanks,
Gene
Reply | Threaded
Open this post in threaded view
|

Re: Shared index base

Matthew Runo
We're about to do the same thing here, but have not tried yet. We  
currently run Solr with replication across several servers. So long as  
only one server is doing updates to the index, I think it should work  
fine.


Thanks!

Matthew Runo
Software Developer
Zappos.com
702.943.7833

On Feb 26, 2008, at 7:51 AM, Evgeniy Strokin wrote:

> I know there was such discussions about the subject, but I want to  
> ask again if somebody could share more information.
> We are planning to have several separate servers for our search  
> engine. One of them will be index/search server, and all others are  
> search only.
> We want to use SAN (BTW: should we consider something else?) and  
> give access to it from all servers. So all servers will use the same  
> index base, without any replication, same files.
> Is this a good practice? Did somebody do the same? Any problems  
> noticed? Or any suggestions, even about different configurations are  
> highly appreciated.
>
> Thanks,
> Gene

Reply | Threaded
Open this post in threaded view
|

Re: Shared index base

Rachel McConnell-3
We tried this architecture for our initial rollout of Solr/Lucene to
our production application.  We ran into a problem with it, which may
or may not apply to you.  Our production software servers all are
monitored for uptime by a daemon which pings them periodically and
restarts them if a response is not received within a configurable
period of time.

We found that under some orderings of restarts, the Lucene appservers
would not come up correctly.  I don't recall the exact details, and I
don't think it ever corrupted the index.  As I recall, we had to
restart in a particular order to avoid freezes on the read-only
servers, and of course the automated monitor, separate for each
server, could not do that.

YMMV of course, but this would be something to test thoroughly in a
shared index situation.  We moved a while ago to each server (even on
the same machine) having its own index files, and using the snapshot
puller/shooter processes for replication.

Rachel

On 2/26/08, Matthew Runo <[hidden email]> wrote:

> We're about to do the same thing here, but have not tried yet. We
>  currently run Solr with replication across several servers. So long as
>  only one server is doing updates to the index, I think it should work
>  fine.
>
>
>  Thanks!
>
>
>  Matthew Runo
>  Software Developer
>  Zappos.com
>  702.943.7833
>
>
>  On Feb 26, 2008, at 7:51 AM, Evgeniy Strokin wrote:
>
>  > I know there was such discussions about the subject, but I want to
>  > ask again if somebody could share more information.
>  > We are planning to have several separate servers for our search
>  > engine. One of them will be index/search server, and all others are
>  > search only.
>  > We want to use SAN (BTW: should we consider something else?) and
>  > give access to it from all servers. So all servers will use the same
>  > index base, without any replication, same files.
>  > Is this a good practice? Did somebody do the same? Any problems
>  > noticed? Or any suggestions, even about different configurations are
>  > highly appreciated.
>  >
>  > Thanks,
>  > Gene
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Shared index base

Alok Dhir
In reply to this post by Matthew Runo
Are you saying all the servers will use the same 'data' dir?  Is that  
a supported config?

On Feb 26, 2008, at 12:29 PM, Matthew Runo wrote:

> We're about to do the same thing here, but have not tried yet. We  
> currently run Solr with replication across several servers. So long  
> as only one server is doing updates to the index, I think it should  
> work fine.
>
>
> Thanks!
>
> Matthew Runo
> Software Developer
> Zappos.com
> 702.943.7833
>
> On Feb 26, 2008, at 7:51 AM, Evgeniy Strokin wrote:
>
>> I know there was such discussions about the subject, but I want to  
>> ask again if somebody could share more information.
>> We are planning to have several separate servers for our search  
>> engine. One of them will be index/search server, and all others are  
>> search only.
>> We want to use SAN (BTW: should we consider something else?) and  
>> give access to it from all servers. So all servers will use the  
>> same index base, without any replication, same files.
>> Is this a good practice? Did somebody do the same? Any problems  
>> noticed? Or any suggestions, even about different configurations  
>> are highly appreciated.
>>
>> Thanks,
>> Gene
>

Reply | Threaded
Open this post in threaded view
|

Re: Shared index base

Matthew Runo
I hope so. I've found that every once in a while Solr 1.2 replication  
will die, from a temp-index.... file that seems to ham it up. Removing  
that file on all the servers fixes the issue though.

We'd like to be able to point all the servers at an NFS location for  
their index files, and use a single server to update it.

Thanks!

Matthew Runo
Software Developer
Zappos.com
702.943.7833

On Feb 26, 2008, at 9:39 AM, Alok Dhir wrote:

> Are you saying all the servers will use the same 'data' dir?  Is  
> that a supported config?
>
> On Feb 26, 2008, at 12:29 PM, Matthew Runo wrote:
>
>> We're about to do the same thing here, but have not tried yet. We  
>> currently run Solr with replication across several servers. So long  
>> as only one server is doing updates to the index, I think it should  
>> work fine.
>>
>>
>> Thanks!
>>
>> Matthew Runo
>> Software Developer
>> Zappos.com
>> 702.943.7833
>>
>> On Feb 26, 2008, at 7:51 AM, Evgeniy Strokin wrote:
>>
>>> I know there was such discussions about the subject, but I want to  
>>> ask again if somebody could share more information.
>>> We are planning to have several separate servers for our search  
>>> engine. One of them will be index/search server, and all others  
>>> are search only.
>>> We want to use SAN (BTW: should we consider something else?) and  
>>> give access to it from all servers. So all servers will use the  
>>> same index base, without any replication, same files.
>>> Is this a good practice? Did somebody do the same? Any problems  
>>> noticed? Or any suggestions, even about different configurations  
>>> are highly appreciated.
>>>
>>> Thanks,
>>> Gene
>>
>

Reply | Threaded
Open this post in threaded view
|

RE: Shared index base

Charlie Jackson
How do you handle commits to the index? By that, I mean that Solr
recreates its searcher when you issue a commit, but only for the system
that does the commit. Wouldn't you be left with searchers on the other
machines that are stale?

- Charlie


-----Original Message-----
From: Matthew Runo [mailto:[hidden email]]
Sent: Tuesday, February 26, 2008 12:18 PM
To: [hidden email]
Subject: Re: Shared index base

I hope so. I've found that every once in a while Solr 1.2 replication  
will die, from a temp-index.... file that seems to ham it up. Removing  
that file on all the servers fixes the issue though.

We'd like to be able to point all the servers at an NFS location for  
their index files, and use a single server to update it.

Thanks!

Matthew Runo
Software Developer
Zappos.com
702.943.7833

On Feb 26, 2008, at 9:39 AM, Alok Dhir wrote:

> Are you saying all the servers will use the same 'data' dir?  Is  
> that a supported config?
>
> On Feb 26, 2008, at 12:29 PM, Matthew Runo wrote:
>
>> We're about to do the same thing here, but have not tried yet. We  
>> currently run Solr with replication across several servers. So long  
>> as only one server is doing updates to the index, I think it should  
>> work fine.
>>
>>
>> Thanks!
>>
>> Matthew Runo
>> Software Developer
>> Zappos.com
>> 702.943.7833
>>
>> On Feb 26, 2008, at 7:51 AM, Evgeniy Strokin wrote:
>>
>>> I know there was such discussions about the subject, but I want to  
>>> ask again if somebody could share more information.
>>> We are planning to have several separate servers for our search  
>>> engine. One of them will be index/search server, and all others  
>>> are search only.
>>> We want to use SAN (BTW: should we consider something else?) and  
>>> give access to it from all servers. So all servers will use the  
>>> same index base, without any replication, same files.
>>> Is this a good practice? Did somebody do the same? Any problems  
>>> noticed? Or any suggestions, even about different configurations  
>>> are highly appreciated.
>>>
>>> Thanks,
>>> Gene
>>
>

Reply | Threaded
Open this post in threaded view
|

Re: Shared index base

Walter Underwood, Netflix
In reply to this post by Matthew Runo
I saw a 100X slowdown running with indexes on NFS.

I don't understand going through a lot of effort with unsupported
configurations just to share an index. Local disk is cheap, the
snapshot stuff works well, and local discs avoid a single point
of failure.

The testing time to make a shared index work with each new
release of Solr is almost certainly more expensive than buying
local disc.

The single point of failure is real issue. I've seen two discs
fail on one RAID. When that happens, you've lost all of your
search for hours or days.

Finally, how do you tell Solr that the index has changed and
it needs a new Searcher? Normally, that is a commit, but you
don't want to commit from a read-only Solr.

wunder

On 2/26/08 10:17 AM, "Matthew Runo" <[hidden email]> wrote:

> I hope so. I've found that every once in a while Solr 1.2 replication
> will die, from a temp-index.... file that seems to ham it up. Removing
> that file on all the servers fixes the issue though.
>
> We'd like to be able to point all the servers at an NFS location for
> their index files, and use a single server to update it.
>
> Thanks!
>
> Matthew Runo
> Software Developer
> Zappos.com
> 702.943.7833
>
> On Feb 26, 2008, at 9:39 AM, Alok Dhir wrote:
>
>> Are you saying all the servers will use the same 'data' dir?  Is
>> that a supported config?
>>
>> On Feb 26, 2008, at 12:29 PM, Matthew Runo wrote:
>>
>>> We're about to do the same thing here, but have not tried yet. We
>>> currently run Solr with replication across several servers. So long
>>> as only one server is doing updates to the index, I think it should
>>> work fine.
>>>
>>>
>>> Thanks!
>>>
>>> Matthew Runo
>>> Software Developer
>>> Zappos.com
>>> 702.943.7833
>>>
>>> On Feb 26, 2008, at 7:51 AM, Evgeniy Strokin wrote:
>>>
>>>> I know there was such discussions about the subject, but I want to
>>>> ask again if somebody could share more information.
>>>> We are planning to have several separate servers for our search
>>>> engine. One of them will be index/search server, and all others
>>>> are search only.
>>>> We want to use SAN (BTW: should we consider something else?) and
>>>> give access to it from all servers. So all servers will use the
>>>> same index base, without any replication, same files.
>>>> Is this a good practice? Did somebody do the same? Any problems
>>>> noticed? Or any suggestions, even about different configurations
>>>> are highly appreciated.
>>>>
>>>> Thanks,
>>>> Gene
>>>
>>
>

Reply | Threaded
Open this post in threaded view
|

RE: Shared index base

Jae Joo-2
In reply to this post by Evgeniy Strokin

In my environment, there is NO big difference between local disk and SAN based file system.
A little slow down, but not a problem (1 or 2 %)
I do have 4 sets of solr indices each has more than 10G in 3 servers.
I think that it is not good way to share SINGLE Index. - disk is pretty cheap and we can add more disk in SAN pretty easily.
I have another server which is called "Master" with local disk based Solr Index to update the index.
By some accident or time out, the update is not done successfully, so I do need to do something by manually.
If you have only one index, there is a risk to mess up the index.

Thanks,

Jae


-----Original Message-----
From: Walter Underwood [mailto:[hidden email]]
Sent: Tue 2/26/2008 1:27 PM
To: [hidden email]
Subject: Re: Shared index base
 
I saw a 100X slowdown running with indexes on NFS.

I don't understand going through a lot of effort with unsupported
configurations just to share an index. Local disk is cheap, the
snapshot stuff works well, and local discs avoid a single point
of failure.

The testing time to make a shared index work with each new
release of Solr is almost certainly more expensive than buying
local disc.

The single point of failure is real issue. I've seen two discs
fail on one RAID. When that happens, you've lost all of your
search for hours or days.

Finally, how do you tell Solr that the index has changed and
it needs a new Searcher? Normally, that is a commit, but you
don't want to commit from a read-only Solr.

wunder

On 2/26/08 10:17 AM, "Matthew Runo" <[hidden email]> wrote:

> I hope so. I've found that every once in a while Solr 1.2 replication
> will die, from a temp-index.... file that seems to ham it up. Removing
> that file on all the servers fixes the issue though.
>
> We'd like to be able to point all the servers at an NFS location for
> their index files, and use a single server to update it.
>
> Thanks!
>
> Matthew Runo
> Software Developer
> Zappos.com
> 702.943.7833
>
> On Feb 26, 2008, at 9:39 AM, Alok Dhir wrote:
>
>> Are you saying all the servers will use the same 'data' dir?  Is
>> that a supported config?
>>
>> On Feb 26, 2008, at 12:29 PM, Matthew Runo wrote:
>>
>>> We're about to do the same thing here, but have not tried yet. We
>>> currently run Solr with replication across several servers. So long
>>> as only one server is doing updates to the index, I think it should
>>> work fine.
>>>
>>>
>>> Thanks!
>>>
>>> Matthew Runo
>>> Software Developer
>>> Zappos.com
>>> 702.943.7833
>>>
>>> On Feb 26, 2008, at 7:51 AM, Evgeniy Strokin wrote:
>>>
>>>> I know there was such discussions about the subject, but I want to
>>>> ask again if somebody could share more information.
>>>> We are planning to have several separate servers for our search
>>>> engine. One of them will be index/search server, and all others
>>>> are search only.
>>>> We want to use SAN (BTW: should we consider something else?) and
>>>> give access to it from all servers. So all servers will use the
>>>> same index base, without any replication, same files.
>>>> Is this a good practice? Did somebody do the same? Any problems
>>>> noticed? Or any suggestions, even about different configurations
>>>> are highly appreciated.
>>>>
>>>> Thanks,
>>>> Gene
>>>
>>
>




Reply | Threaded
Open this post in threaded view
|

Re: Shared index base

Walter Underwood, Netflix
SAN is not NFS. I would expect SAN to be fast.

wunder

On 2/26/08 10:47 AM, "Jae Joo" <[hidden email]> wrote:

>
> In my environment, there is NO big difference between local disk and SAN based
> file system.
> A little slow down, but not a problem (1 or 2 %)
> I do have 4 sets of solr indices each has more than 10G in 3 servers.
> I think that it is not good way to share SINGLE Index. - disk is pretty cheap
> and we can add more disk in SAN pretty easily.
> I have another server which is called "Master" with local disk based Solr
> Index to update the index.
> By some accident or time out, the update is not done successfully, so I do
> need to do something by manually.
> If you have only one index, there is a risk to mess up the index.
>
> Thanks,
>
> Jae
>
>
> -----Original Message-----
> From: Walter Underwood [mailto:[hidden email]]
> Sent: Tue 2/26/2008 1:27 PM
> To: [hidden email]
> Subject: Re: Shared index base
>  
> I saw a 100X slowdown running with indexes on NFS.
>
> I don't understand going through a lot of effort with unsupported
> configurations just to share an index. Local disk is cheap, the
> snapshot stuff works well, and local discs avoid a single point
> of failure.
>
> The testing time to make a shared index work with each new
> release of Solr is almost certainly more expensive than buying
> local disc.
>
> The single point of failure is real issue. I've seen two discs
> fail on one RAID. When that happens, you've lost all of your
> search for hours or days.
>
> Finally, how do you tell Solr that the index has changed and
> it needs a new Searcher? Normally, that is a commit, but you
> don't want to commit from a read-only Solr.
>
> wunder
>
> On 2/26/08 10:17 AM, "Matthew Runo" <[hidden email]> wrote:
>
>> I hope so. I've found that every once in a while Solr 1.2 replication
>> will die, from a temp-index.... file that seems to ham it up. Removing
>> that file on all the servers fixes the issue though.
>>
>> We'd like to be able to point all the servers at an NFS location for
>> their index files, and use a single server to update it.
>>
>> Thanks!
>>
>> Matthew Runo
>> Software Developer
>> Zappos.com
>> 702.943.7833
>>
>> On Feb 26, 2008, at 9:39 AM, Alok Dhir wrote:
>>
>>> Are you saying all the servers will use the same 'data' dir?  Is
>>> that a supported config?
>>>
>>> On Feb 26, 2008, at 12:29 PM, Matthew Runo wrote:
>>>
>>>> We're about to do the same thing here, but have not tried yet. We
>>>> currently run Solr with replication across several servers. So long
>>>> as only one server is doing updates to the index, I think it should
>>>> work fine.
>>>>
>>>>
>>>> Thanks!
>>>>
>>>> Matthew Runo
>>>> Software Developer
>>>> Zappos.com
>>>> 702.943.7833
>>>>
>>>> On Feb 26, 2008, at 7:51 AM, Evgeniy Strokin wrote:
>>>>
>>>>> I know there was such discussions about the subject, but I want to
>>>>> ask again if somebody could share more information.
>>>>> We are planning to have several separate servers for our search
>>>>> engine. One of them will be index/search server, and all others
>>>>> are search only.
>>>>> We want to use SAN (BTW: should we consider something else?) and
>>>>> give access to it from all servers. So all servers will use the
>>>>> same index base, without any replication, same files.
>>>>> Is this a good practice? Did somebody do the same? Any problems
>>>>> noticed? Or any suggestions, even about different configurations
>>>>> are highly appreciated.
>>>>>
>>>>> Thanks,
>>>>> Gene
>>>>
>>>
>>
>
>
>
>

Reply | Threaded
Open this post in threaded view
|

Re: Shared index base

Matthew Runo
That's true about the commit issue. With that in mind, it might be  
better to use replication - just keep an eye on it to ensure it's  
working, as my 1.2 install (3 servers) tends to stop every once in a  
blue moon.

Thanks!

Matthew Runo
Software Developer
Zappos.com
702.943.7833

On Feb 26, 2008, at 10:53 AM, Walter Underwood wrote:

> SAN is not NFS. I would expect SAN to be fast.
>
> wunder
>
> On 2/26/08 10:47 AM, "Jae Joo" <[hidden email]> wrote:
>
>>
>> In my environment, there is NO big difference between local disk  
>> and SAN based
>> file system.
>> A little slow down, but not a problem (1 or 2 %)
>> I do have 4 sets of solr indices each has more than 10G in 3 servers.
>> I think that it is not good way to share SINGLE Index. - disk is  
>> pretty cheap
>> and we can add more disk in SAN pretty easily.
>> I have another server which is called "Master" with local disk  
>> based Solr
>> Index to update the index.
>> By some accident or time out, the update is not done successfully,  
>> so I do
>> need to do something by manually.
>> If you have only one index, there is a risk to mess up the index.
>>
>> Thanks,
>>
>> Jae
>>
>>
>> -----Original Message-----
>> From: Walter Underwood [mailto:[hidden email]]
>> Sent: Tue 2/26/2008 1:27 PM
>> To: [hidden email]
>> Subject: Re: Shared index base
>>
>> I saw a 100X slowdown running with indexes on NFS.
>>
>> I don't understand going through a lot of effort with unsupported
>> configurations just to share an index. Local disk is cheap, the
>> snapshot stuff works well, and local discs avoid a single point
>> of failure.
>>
>> The testing time to make a shared index work with each new
>> release of Solr is almost certainly more expensive than buying
>> local disc.
>>
>> The single point of failure is real issue. I've seen two discs
>> fail on one RAID. When that happens, you've lost all of your
>> search for hours or days.
>>
>> Finally, how do you tell Solr that the index has changed and
>> it needs a new Searcher? Normally, that is a commit, but you
>> don't want to commit from a read-only Solr.
>>
>> wunder
>>
>> On 2/26/08 10:17 AM, "Matthew Runo" <[hidden email]> wrote:
>>
>>> I hope so. I've found that every once in a while Solr 1.2  
>>> replication
>>> will die, from a temp-index.... file that seems to ham it up.  
>>> Removing
>>> that file on all the servers fixes the issue though.
>>>
>>> We'd like to be able to point all the servers at an NFS location for
>>> their index files, and use a single server to update it.
>>>
>>> Thanks!
>>>
>>> Matthew Runo
>>> Software Developer
>>> Zappos.com
>>> 702.943.7833
>>>
>>> On Feb 26, 2008, at 9:39 AM, Alok Dhir wrote:
>>>
>>>> Are you saying all the servers will use the same 'data' dir?  Is
>>>> that a supported config?
>>>>
>>>> On Feb 26, 2008, at 12:29 PM, Matthew Runo wrote:
>>>>
>>>>> We're about to do the same thing here, but have not tried yet. We
>>>>> currently run Solr with replication across several servers. So  
>>>>> long
>>>>> as only one server is doing updates to the index, I think it  
>>>>> should
>>>>> work fine.
>>>>>
>>>>>
>>>>> Thanks!
>>>>>
>>>>> Matthew Runo
>>>>> Software Developer
>>>>> Zappos.com
>>>>> 702.943.7833
>>>>>
>>>>> On Feb 26, 2008, at 7:51 AM, Evgeniy Strokin wrote:
>>>>>
>>>>>> I know there was such discussions about the subject, but I want  
>>>>>> to
>>>>>> ask again if somebody could share more information.
>>>>>> We are planning to have several separate servers for our search
>>>>>> engine. One of them will be index/search server, and all others
>>>>>> are search only.
>>>>>> We want to use SAN (BTW: should we consider something else?) and
>>>>>> give access to it from all servers. So all servers will use the
>>>>>> same index base, without any replication, same files.
>>>>>> Is this a good practice? Did somebody do the same? Any problems
>>>>>> noticed? Or any suggestions, even about different configurations
>>>>>> are highly appreciated.
>>>>>>
>>>>>> Thanks,
>>>>>> Gene
>>>>>
>>>>
>>>
>>
>>
>>
>>
>

Reply | Threaded
Open this post in threaded view
|

Re: Shared index base

Mike Klaas
In reply to this post by Alok Dhir
There hasn't really been a concrete answer given in this thread, so:  
It works to point multiple Solr's at a single data dir, but you can't  
have more than one writer.  If you try, the index could become  
corrupted or inconsistent (especially if you are using 'simple' lock  
type).  Also, the Solrs do not communicate with each other.  You have  
to tell the readers manually that the index is updated (via commit()--
autoCommit will not work).

-Mike

On 26-Feb-08, at 9:39 AM, Alok Dhir wrote:

> Are you saying all the servers will use the same 'data' dir?  Is  
> that a supported config?
>
> On Feb 26, 2008, at 12:29 PM, Matthew Runo wrote:
>
>> We're about to do the same thing here, but have not tried yet. We  
>> currently run Solr with replication across several servers. So long  
>> as only one server is doing updates to the index, I think it should  
>> work fine.
>>
>>
>> Thanks!
>>
>> Matthew Runo
>> Software Developer
>> Zappos.com
>> 702.943.7833
>>
>> On Feb 26, 2008, at 7:51 AM, Evgeniy Strokin wrote:
>>
>>> I know there was such discussions about the subject, but I want to  
>>> ask again if somebody could share more information.
>>> We are planning to have several separate servers for our search  
>>> engine. One of them will be index/search server, and all others  
>>> are search only.
>>> We want to use SAN (BTW: should we consider something else?) and  
>>> give access to it from all servers. So all servers will use the  
>>> same index base, without any replication, same files.
>>> Is this a good practice? Did somebody do the same? Any problems  
>>> noticed? Or any suggestions, even about different configurations  
>>> are highly appreciated.
>>>
>>> Thanks,
>>> Gene
>>
>

Reply | Threaded
Open this post in threaded view
|

Re: Shared index base

Alok Dhir
thanks for your response - i've been waiting for this very  
clarification.  so 'commit()' makes readers re-read the indexes?
On Feb 26, 2008, at 7:03 PM, Mike Klaas wrote:

> There hasn't really been a concrete answer given in this thread,  
> so:  It works to point multiple Solr's at a single data dir, but you  
> can't have more than one writer.  If you try, the index could become  
> corrupted or inconsistent (especially if you are using 'simple' lock  
> type).  Also, the Solrs do not communicate with each other.  You  
> have to tell the readers manually that the index is updated (via  
> commit()--autoCommit will not work).
>
> -Mike
>
> On 26-Feb-08, at 9:39 AM, Alok Dhir wrote:
>
>> Are you saying all the servers will use the same 'data' dir?  Is  
>> that a supported config?
>>
>> On Feb 26, 2008, at 12:29 PM, Matthew Runo wrote:
>>
>>> We're about to do the same thing here, but have not tried yet. We  
>>> currently run Solr with replication across several servers. So  
>>> long as only one server is doing updates to the index, I think it  
>>> should work fine.
>>>
>>>
>>> Thanks!
>>>
>>> Matthew Runo
>>> Software Developer
>>> Zappos.com
>>> 702.943.7833
>>>
>>> On Feb 26, 2008, at 7:51 AM, Evgeniy Strokin wrote:
>>>
>>>> I know there was such discussions about the subject, but I want  
>>>> to ask again if somebody could share more information.
>>>> We are planning to have several separate servers for our search  
>>>> engine. One of them will be index/search server, and all others  
>>>> are search only.
>>>> We want to use SAN (BTW: should we consider something else?) and  
>>>> give access to it from all servers. So all servers will use the  
>>>> same index base, without any replication, same files.
>>>> Is this a good practice? Did somebody do the same? Any problems  
>>>> noticed? Or any suggestions, even about different configurations  
>>>> are highly appreciated.
>>>>
>>>> Thanks,
>>>> Gene
>>>
>>
>

Reply | Threaded
Open this post in threaded view
|

Re: Shared index base

Otis Gospodnetic-2
In reply to this post by Evgeniy Strokin
Alok: correct - commit causes Solr to re-open the index.

Gene: That should work just fine.  While you can't have multiple concurrent writers, you can send multiple concurrent indexing requests to a single Solr instance designated to be the master.

Otis

--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch

----- Original Message ----

> From: Alok K. Dhir <[hidden email]>
> To: [hidden email]
> Sent: Tuesday, February 26, 2008 7:51:19 PM
> Subject: Re: Shared index base
>
> thanks for your response - i've been waiting for this very  
> clarification.  so 'commit()' makes readers re-read the indexes?
> On Feb 26, 2008, at 7:03 PM, Mike Klaas wrote:
>
> > There hasn't really been a concrete answer given in this thread,  
> > so:  It works to point multiple Solr's at a single data dir, but you  
> > can't have more than one writer.  If you try, the index could become  
> > corrupted or inconsistent (especially if you are using 'simple' lock  
> > type).  Also, the Solrs do not communicate with each other.  You  
> > have to tell the readers manually that the index is updated (via  
> > commit()--autoCommit will not work).
> >
> > -Mike
> >
> > On 26-Feb-08, at 9:39 AM, Alok Dhir wrote:
> >
> >> Are you saying all the servers will use the same 'data' dir?  Is  
> >> that a supported config?
> >>
> >> On Feb 26, 2008, at 12:29 PM, Matthew Runo wrote:
> >>
> >>> We're about to do the same thing here, but have not tried yet. We  
> >>> currently run Solr with replication across several servers. So  
> >>> long as only one server is doing updates to the index, I think it  
> >>> should work fine.
> >>>
> >>>
> >>> Thanks!
> >>>
> >>> Matthew Runo
> >>> Software Developer
> >>> Zappos.com
> >>> 702.943.7833
> >>>
> >>> On Feb 26, 2008, at 7:51 AM, Evgeniy Strokin wrote:
> >>>
> >>>> I know there was such discussions about the subject, but I want  
> >>>> to ask again if somebody could share more information.
> >>>> We are planning to have several separate servers for our search  
> >>>> engine. One of them will be index/search server, and all others  
> >>>> are search only.
> >>>> We want to use SAN (BTW: should we consider something else?) and  
> >>>> give access to it from all servers. So all servers will use the  
> >>>> same index base, without any replication, same files.
> >>>> Is this a good practice? Did somebody do the same? Any problems  
> >>>> noticed? Or any suggestions, even about different configurations  
> >>>> are highly appreciated.
> >>>>
> >>>> Thanks,
> >>>> Gene
> >>>
> >>
> >
>
>


Reply | Threaded
Open this post in threaded view
|

Re: Shared index base

Alok Dhir
Here's another question on this rather old thread -- while poring  
through various options in solrconfig, I came across the the 'native'  
lockType option.

That seems to indicate that SOLR/Lucene should work fine with multiple  
writers, as long as a proper locking mechanism is in place, such as  
would be provided by a POSIX compliant cluster file system, such as  
GPFS, GFS, Ibrix, OCFS2...

Single shared index, multiple readers/writers, as long as the  
underlying filesystem implements fs locks properly.

Is this correct?

---
Alok K. Dhir
[hidden email]
Symplicity Corporation
1 703 351 0200 x 8080
www.symplicity.com

On Feb 27, 2008, at 3:10 AM, Otis Gospodnetic wrote:

> Alok: correct - commit causes Solr to re-open the index.
>
> Gene: That should work just fine.  While you can't have multiple  
> concurrent writers, you can send multiple concurrent indexing  
> requests to a single Solr instance designated to be the master.
>
> Otis
>
> --
> Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
>
> ----- Original Message ----
>> From: Alok K. Dhir <[hidden email]>
>> To: [hidden email]
>> Sent: Tuesday, February 26, 2008 7:51:19 PM
>> Subject: Re: Shared index base
>>
>> thanks for your response - i've been waiting for this very
>> clarification.  so 'commit()' makes readers re-read the indexes?
>> On Feb 26, 2008, at 7:03 PM, Mike Klaas wrote:
>>
>>> There hasn't really been a concrete answer given in this thread,
>>> so:  It works to point multiple Solr's at a single data dir, but you
>>> can't have more than one writer.  If you try, the index could become
>>> corrupted or inconsistent (especially if you are using 'simple' lock
>>> type).  Also, the Solrs do not communicate with each other.  You
>>> have to tell the readers manually that the index is updated (via
>>> commit()--autoCommit will not work).
>>>
>>> -Mike
>>>
>>> On 26-Feb-08, at 9:39 AM, Alok Dhir wrote:
>>>
>>>> Are you saying all the servers will use the same 'data' dir?  Is
>>>> that a supported config?
>>>>
>>>> On Feb 26, 2008, at 12:29 PM, Matthew Runo wrote:
>>>>
>>>>> We're about to do the same thing here, but have not tried yet. We
>>>>> currently run Solr with replication across several servers. So
>>>>> long as only one server is doing updates to the index, I think it
>>>>> should work fine.
>>>>>
>>>>>
>>>>> Thanks!
>>>>>
>>>>> Matthew Runo
>>>>> Software Developer
>>>>> Zappos.com
>>>>> 702.943.7833
>>>>>
>>>>> On Feb 26, 2008, at 7:51 AM, Evgeniy Strokin wrote:
>>>>>
>>>>>> I know there was such discussions about the subject, but I want
>>>>>> to ask again if somebody could share more information.
>>>>>> We are planning to have several separate servers for our search
>>>>>> engine. One of them will be index/search server, and all others
>>>>>> are search only.
>>>>>> We want to use SAN (BTW: should we consider something else?) and
>>>>>> give access to it from all servers. So all servers will use the
>>>>>> same index base, without any replication, same files.
>>>>>> Is this a good practice? Did somebody do the same? Any problems
>>>>>> noticed? Or any suggestions, even about different configurations
>>>>>> are highly appreciated.
>>>>>>
>>>>>> Thanks,
>>>>>> Gene
>>>>>
>>>>
>>>
>>
>>
>
>

Reply | Threaded
Open this post in threaded view
|

Re: Shared index base

Mike Klaas

On 2-May-08, at 1:20 PM, Alok Dhir wrote:

> Here's another question on this rather old thread -- while poring  
> through various options in solrconfig, I came across the the  
> 'native' lockType option.
>
> That seems to indicate that SOLR/Lucene should work fine with  
> multiple writers, as long as a proper locking mechanism is in place,  
> such as would be provided by a POSIX compliant cluster file system,  
> such as GPFS, GFS, Ibrix, OCFS2...
>
> Single shared index, multiple readers/writers, as long as the  
> underlying filesystem implements fs locks properly.
>
> Is this correct?

No.  You wll avoid index corruption, but deletions/updates may not be  
handled properly.

-Mike