I am also in the same idea. Based on the field, I could shard but there are two practical difficulties.
1. If normal user logged-in then result could be fetched from the corresponding search server but if Admin user logged-in, then he may need to see all data. The query should be issued across servers and results should be consolidated.
2. Consider a scenario I am sharding based on the User, I am having single search server and It is handling 1000 members. Now as the memory consumption is high, I have added one more search server. New users could access the second server but what about the old users, their data will be still added to the server1. How to address this issue. Is rebuilding the index the only way.
Could any one share their experience, How they solved scale out problems?
----- Original Message -----
From: "Anshum" <[hidden email]>
To: <[hidden email]>
Sent: Friday, January 21, 2011 12:04 PM
Subject: Re: Scale out design patterns
> Hi Ganesh,
> I'd suggest, if you have a particular dimension/field on which you could
> shard your data such that the query/data breakup gets predictable, that
> would be a good way to scale out e.g. if you have users which are equally
> active/searched then you may want to split their data on a simple mod of
> some numeric (auto increment) userid.
> This works well under normal cases unless your partitioning is not
> Anshum Gupta
> http://ai-cafe.blogspot.com >
> On Fri, Jan 21, 2011 at 10:52 AM, Ganesh <[hidden email]> wrote:
>> Hello all,
>> Could you any one guide me what all the various ways we could scale out?
>> 1. Index: Add data to the nodes in round-robin.
>> Search: Query all the nodes and cluster the results using carrot2.
>> 2.Horizontal partitioning and No shared architecture,
>> Index: Split the data based on userid and index few set of users data
>> in each node.
>> Search: Have a mapper kind of application which could tell which userid
>> is mapped to node, redirect the search traffic to corresponding node.
>> Which one is best? Did you guys tried any of these approach. Please share
>> your thoughts.
>> Send free SMS to your Friends on Mobile from your Yahoo! Messenger.
>> Download Now! http://messenger.yahoo.com/download.php >>
>> To unsubscribe, e-mail: [hidden email] >> For additional commands, e-mail: [hidden email] >>