consistency in SolrCloud replication

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

consistency in SolrCloud replication

Bill Au
Will a newly indexed document included in search result in the shard leader
as soon as it has been indexed locally or is it included in search result
only after it has been forwarded to and indexed in all the replicas?

Bill
Reply | Threaded
Open this post in threaded view
|

Re: consistency in SolrCloud replication

Mark Miller-3
It's included as soon as it has been indexed - though a request won't return until it's affected all replicas. Low latency eventual consistency.

- Mark

On Nov 14, 2012, at 5:47 PM, Bill Au <[hidden email]> wrote:

> Will a newly indexed document included in search result in the shard leader
> as soon as it has been indexed locally or is it included in search result
> only after it has been forwarded to and indexed in all the replicas?
>
> Bill

Reply | Threaded
Open this post in threaded view
|

Re: consistency in SolrCloud replication

Bill Au
Thanks for the info, Mark.  By "a request won't return until it's affected
all replicas", are you referring to the update request or the query?

Bill


On Wed, Nov 14, 2012 at 7:57 PM, Mark Miller <[hidden email]> wrote:

> It's included as soon as it has been indexed - though a request won't
> return until it's affected all replicas. Low latency eventual consistency.
>
> - Mark
>
> On Nov 14, 2012, at 5:47 PM, Bill Au <[hidden email]> wrote:
>
> > Will a newly indexed document included in search result in the shard
> leader
> > as soon as it has been indexed locally or is it included in search result
> > only after it has been forwarded to and indexed in all the replicas?
> >
> > Bill
>
>
Reply | Threaded
Open this post in threaded view
|

Re: consistency in SolrCloud replication

Mark Miller-3
I'm talking about an update request. So if you make an update, when it
returns, your next search will see the update, because it will be on
all replicas. Another process that is searching rapidly may see an
"eventually" consistent view though (very briefly). We have some ideas
to make that view "more consistent", for example by allowing for
searcher leases.

- Mark

On Thu, Nov 15, 2012 at 8:53 AM, Bill Au <[hidden email]> wrote:

> Thanks for the info, Mark.  By "a request won't return until it's affected
> all replicas", are you referring to the update request or the query?
>
> Bill
>
>
> On Wed, Nov 14, 2012 at 7:57 PM, Mark Miller <[hidden email]> wrote:
>
>> It's included as soon as it has been indexed - though a request won't
>> return until it's affected all replicas. Low latency eventual consistency.
>>
>> - Mark
>>
>> On Nov 14, 2012, at 5:47 PM, Bill Au <[hidden email]> wrote:
>>
>> > Will a newly indexed document included in search result in the shard
>> leader
>> > as soon as it has been indexed locally or is it included in search result
>> > only after it has been forwarded to and indexed in all the replicas?
>> >
>> > Bill
>>
>>



--
- Mark
Reply | Threaded
Open this post in threaded view
|

Re: consistency in SolrCloud replication

David Smiley
Mark Miller-3 wrote
I'm talking about an update request. So if you make an update, when it
returns, your next search will see the update, because it will be on
all replicas.
I presume this is only the case if (of course) the client also sent a commit.  So you're saying the commit call will not return unless all replicas have completed their commits.  Right?

~ David
Reply | Threaded
Open this post in threaded view
|

Re: consistency in SolrCloud replication

Mark Miller-3
It depends - no commit necessary for realtime get. Otherwise, yes, you would need to do at least a soft commit. That works the same way though - so if you make your update, then do a soft commit, you can be sure your next search will see the update on all the replicas. And with realtime get, of course no commit is necessary to see it.

- Mark

On Nov 15, 2012, at 10:40 AM, David Smiley (@MITRE.org) <[hidden email]> wrote:

> Mark Miller-3 wrote
>> I'm talking about an update request. So if you make an update, when it
>> returns, your next search will see the update, because it will be on
>> all replicas.
>
> I presume this is only the case if (of course) the client also sent a
> commit.  So you're saying the commit call will not return unless all
> replicas have completed their commits.  Right?
>
> ~ David
>
>
>
> -----
> Author: http://www.packtpub.com/apache-solr-3-enterprise-search-server/book
> --
> View this message in context: http://lucene.472066.n3.nabble.com/consistency-in-SolrCloud-replication-tp4020379p4020518.html
> Sent from the Solr - User mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|

Re: consistency in SolrCloud replication

Otis Gospodnetić
I think Bill was asking about search....
I think the Q is whether the query hitting the shard where a doc was sent
for indexing would see that doc even before that doc has been copied to
replicas.

I didn't test it, but I'd think the answer would be positive because of the
xa log.

Otis
--
Performance Monitoring - http://sematext.com/spm
On Nov 15, 2012 11:30 AM, "Mark Miller" <[hidden email]> wrote:

> It depends - no commit necessary for realtime get. Otherwise, yes, you
> would need to do at least a soft commit. That works the same way though -
> so if you make your update, then do a soft commit, you can be sure your
> next search will see the update on all the replicas. And with realtime get,
> of course no commit is necessary to see it.
>
> - Mark
>
> On Nov 15, 2012, at 10:40 AM, David Smiley (@MITRE.org) <[hidden email]>
> wrote:
>
> > Mark Miller-3 wrote
> >> I'm talking about an update request. So if you make an update, when it
> >> returns, your next search will see the update, because it will be on
> >> all replicas.
> >
> > I presume this is only the case if (of course) the client also sent a
> > commit.  So you're saying the commit call will not return unless all
> > replicas have completed their commits.  Right?
> >
> > ~ David
> >
> >
> >
> > -----
> > Author:
> http://www.packtpub.com/apache-solr-3-enterprise-search-server/book
> > --
> > View this message in context:
> http://lucene.472066.n3.nabble.com/consistency-in-SolrCloud-replication-tp4020379p4020518.html
> > Sent from the Solr - User mailing list archive at Nabble.com.
>
>
Reply | Threaded
Open this post in threaded view
|

Re: consistency in SolrCloud replication

Bill Au
Yes, my original question is about search.  And Mark did answered is in his
original reply.  I am guessing that the replicas are updated sequentially
so the newly added documents will be available in some replicas before
other.  I want to know where SolrCloud stands in terms of CAP.

Bill


On Thu, Nov 15, 2012 at 10:31 PM, Otis Gospodnetic <
[hidden email]> wrote:

> I think Bill was asking about search....
> I think the Q is whether the query hitting the shard where a doc was sent
> for indexing would see that doc even before that doc has been copied to
> replicas.
>
> I didn't test it, but I'd think the answer would be positive because of the
> xa log.
>
> Otis
> --
> Performance Monitoring - http://sematext.com/spm
> On Nov 15, 2012 11:30 AM, "Mark Miller" <[hidden email]> wrote:
>
> > It depends - no commit necessary for realtime get. Otherwise, yes, you
> > would need to do at least a soft commit. That works the same way though -
> > so if you make your update, then do a soft commit, you can be sure your
> > next search will see the update on all the replicas. And with realtime
> get,
> > of course no commit is necessary to see it.
> >
> > - Mark
> >
> > On Nov 15, 2012, at 10:40 AM, David Smiley (@MITRE.org) <
> [hidden email]>
> > wrote:
> >
> > > Mark Miller-3 wrote
> > >> I'm talking about an update request. So if you make an update, when it
> > >> returns, your next search will see the update, because it will be on
> > >> all replicas.
> > >
> > > I presume this is only the case if (of course) the client also sent a
> > > commit.  So you're saying the commit call will not return unless all
> > > replicas have completed their commits.  Right?
> > >
> > > ~ David
> > >
> > >
> > >
> > > -----
> > > Author:
> > http://www.packtpub.com/apache-solr-3-enterprise-search-server/book
> > > --
> > > View this message in context:
> >
> http://lucene.472066.n3.nabble.com/consistency-in-SolrCloud-replication-tp4020379p4020518.html
> > > Sent from the Solr - User mailing list archive at Nabble.com.
> >
> >
>
Reply | Threaded
Open this post in threaded view
|

RE: consistency in SolrCloud replication

Markus Jelsma-2
Solr is provides availability and it is tolerant to partioning so that leaves consistency. It is eventual consistent.
 
-----Original message-----

> From:Bill Au <[hidden email]>
> Sent: Fri 16-Nov-2012 15:00
> To: [hidden email]
> Subject: Re: consistency in SolrCloud replication
>
> Yes, my original question is about search.  And Mark did answered is in his
> original reply.  I am guessing that the replicas are updated sequentially
> so the newly added documents will be available in some replicas before
> other.  I want to know where SolrCloud stands in terms of CAP.
>
> Bill
>
>
> On Thu, Nov 15, 2012 at 10:31 PM, Otis Gospodnetic <
> [hidden email]> wrote:
>
> > I think Bill was asking about search....
> > I think the Q is whether the query hitting the shard where a doc was sent
> > for indexing would see that doc even before that doc has been copied to
> > replicas.
> >
> > I didn't test it, but I'd think the answer would be positive because of the
> > xa log.
> >
> > Otis
> > --
> > Performance Monitoring - http://sematext.com/spm
> > On Nov 15, 2012 11:30 AM, "Mark Miller" <[hidden email]> wrote:
> >
> > > It depends - no commit necessary for realtime get. Otherwise, yes, you
> > > would need to do at least a soft commit. That works the same way though -
> > > so if you make your update, then do a soft commit, you can be sure your
> > > next search will see the update on all the replicas. And with realtime
> > get,
> > > of course no commit is necessary to see it.
> > >
> > > - Mark
> > >
> > > On Nov 15, 2012, at 10:40 AM, David Smiley (@MITRE.org) <
> > [hidden email]>
> > > wrote:
> > >
> > > > Mark Miller-3 wrote
> > > >> I'm talking about an update request. So if you make an update, when it
> > > >> returns, your next search will see the update, because it will be on
> > > >> all replicas.
> > > >
> > > > I presume this is only the case if (of course) the client also sent a
> > > > commit.  So you're saying the commit call will not return unless all
> > > > replicas have completed their commits.  Right?
> > > >
> > > > ~ David
> > > >
> > > >
> > > >
> > > > -----
> > > > Author:
> > > http://www.packtpub.com/apache-solr-3-enterprise-search-server/book
> > > > --
> > > > View this message in context:
> > >
> > http://lucene.472066.n3.nabble.com/consistency-in-SolrCloud-replication-tp4020379p4020518.html
> > > > Sent from the Solr - User mailing list archive at Nabble.com.
> > >
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: consistency in SolrCloud replication

Mark Miller-3
> I want to know where SolrCloud stands in terms of CAP.

SolrCloud is a CP system. In the face of partitions, SolrCloud favors
consistency over availability (mostly concerning writes). The system
is eventually consistent, but should become consistent with a pretty
low latency, unlike many cases with an AP system (dynamo style
systems).

You can get things like read your own writes (simple out of the box
behavior) and monotonic reads with the SolrCloud architecture if your
application requires it.

Hopefully, we will be adding more knobs to control some of the
tradeoffs that the current architecture can make over time.

--
- Mark