Large Scale Searching

[Dennis Kubes]

Is anyone doing large scale searching and if so what kind of
architecture is good.  I have a 25G index now (merged) and the searches
are failing due to memory constraints.  Is is better to have multiple
smaller indexes across machines.  If so are those indexes stored on
local machines or in the DFS?  Also what is the searchservers.txt file
for.  I am just starting to implement large scale searching and am a
little confused.

Dennis

[Stefan Groschupf-2]

Am 12.06.2006 um 19:46 schrieb Dennis Kubes:

> Is anyone doing large scale searching and if so what kind of  
> architecture is good.  I have a 25G index now (merged) and the  
> searches are failing due to memory constraints.  Is is better to  
> have multiple smaller indexes across machines.
yes.
> If so are those indexes stored on local machines or in the DFS?
on local hdds.

> Also what is the searchservers.txt file for.
List the ip : port tuples of the search servers.
>   I am just starting to implement large scale searching and am a  
> little confused.

HTH
Stefan