Newbie question on distributed writes onto SolrCloud

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Newbie question on distributed writes onto SolrCloud

Manuel Gonzalo
Hi all,

while investigating SolrCloud for our distributed search needs  I can't seem to find the means to perform distributed writes/updates ... Distributed access to data is available but from what I understood each node has to be accessed independently in order to write/update a value.

Is this so or am I missing something?

Thanks in advance,


Manuel Gonzalo
Software Engineer

http://recommender.strands.com





---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Newbie question on distributed writes onto SolrCloud

Malcolm Upayavira Holmes


On Fri, 26 Nov 2010 18:56 +0100, "Manuel Gonzalo" <[hidden email]>
wrote:

> Hi all,
>
> while investigating SolrCloud for our distributed search needs  I can't
> seem to find the means to perform distributed writes/updates ...
> Distributed access to data is available but from what I understood each
> node has to be accessed independently in order to write/update a value.
>
> Is this so or am I missing something?
>
> Thanks in advance,

As I understand it, distributed write is a TODO as a part of SolrCloud.

It would require a ShardStrategy interface, and probably a default
interface that simply does a MOD shard_count on the document ID.

Also, it would have to split incoming posts and distrubute amongst
shards (an incoming block of 500 documents would maybe end up as 5 posts
of 100 if we had five shards).

Upayavira

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Newbie question on distributed writes onto SolrCloud

Manuel Gonzalo
Thanks for your answer.

I would be really interested in developing this system for distributed writing. Could you please give me a pair of hints about where I should start looking into the code in order to make these modifications? Or is it something that should be developed completely outside the current code?

Thanks!

On 26/11/2010, at 21:16, Upayavira wrote:

>
>
> On Fri, 26 Nov 2010 18:56 +0100, "Manuel Gonzalo" <[hidden email]>
> wrote:
>> Hi all,
>>
>> while investigating SolrCloud for our distributed search needs  I can't
>> seem to find the means to perform distributed writes/updates ...
>> Distributed access to data is available but from what I understood each
>> node has to be accessed independently in order to write/update a value.
>>
>> Is this so or am I missing something?
>>
>> Thanks in advance,
>
> As I understand it, distributed write is a TODO as a part of SolrCloud.
>
> It would require a ShardStrategy interface, and probably a default
> interface that simply does a MOD shard_count on the document ID.
>
> Also, it would have to split incoming posts and distrubute amongst
> shards (an incoming block of 500 documents would maybe end up as 5 posts
> of 100 if we had five shards).
>
> Upayavira
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

Manuel Gonzalo
Software Engineer

http://recommender.strands.com





---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Newbie question on distributed writes onto SolrCloud

Malcolm Upayavira Holmes
I'm not a solr developer, merely someone who has thought about
implementing some of this stuff one day.

Were I to attempt it, I'd dive in to the existing code that is used to
implement the distributed search. Look at the interfaces that are used,
work out how it manages http requests, etc. Then come up with a
ShardingStrategy interface, and build an initial implementation of that
that follows the patterns I saw when looking at the distributed search
code.

But as I say, I'm not a solr developer at this point, and am happy for
others to chime in with better ideas.

Upayavira

On Mon, 29 Nov 2010 10:57 +0100, "Manuel Gonzalo" <[hidden email]>
wrote:

> Thanks for your answer.
>
> I would be really interested in developing this system for distributed
> writing. Could you please give me a pair of hints about where I should
> start looking into the code in order to make these modifications? Or is
> it something that should be developed completely outside the current
> code?
>
> Thanks!
>
> On 26/11/2010, at 21:16, Upayavira wrote:
>
> >
> >
> > On Fri, 26 Nov 2010 18:56 +0100, "Manuel Gonzalo" <[hidden email]>
> > wrote:
> >> Hi all,
> >>
> >> while investigating SolrCloud for our distributed search needs  I can't
> >> seem to find the means to perform distributed writes/updates ...
> >> Distributed access to data is available but from what I understood each
> >> node has to be accessed independently in order to write/update a value.
> >>
> >> Is this so or am I missing something?
> >>
> >> Thanks in advance,
> >
> > As I understand it, distributed write is a TODO as a part of SolrCloud.
> >
> > It would require a ShardStrategy interface, and probably a default
> > interface that simply does a MOD shard_count on the document ID.
> >
> > Also, it would have to split incoming posts and distrubute amongst
> > shards (an incoming block of 500 documents would maybe end up as 5 posts
> > of 100 if we had five shards).
> >
> > Upayavira
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]
> >
>
> Manuel Gonzalo
> Software Engineer
>
> http://recommender.strands.com
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]