Setting preferred replica for query/read

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Setting preferred replica for query/read

Zheng Lin Edwin Yeo
Hi,

Is there any similarities between these two requests in the JIRA regarding
setting of prefer replica function?

(SOLR-11982) Add support for preferReplicaTypes parameter

(SOLR-8146) Allowing SolrJ CloudSolrClient to have preferred replica for
query/read

I am looking at setting one of the replica to be the preferred replica for
query/read, and another replica to be use for indexing.

I am using Solr 7.3.1 currently.

Regards,
Edwin
Reply | Threaded
Open this post in threaded view
|

Re: Setting preferred replica for query/read

Ere Maijala
Hi,

Well, SOLR-11982 adds server-side support for part of what SOLR-8146
aims to do (shards.preference=replica.location:[something]). It doesn't
do regular expressions or snitches at the moment, though it would be
easy to add. So, it looks to me like SOLR-8146 would need to be updated
in this regard.

--Ere

Zheng Lin Edwin Yeo kirjoitti 4.6.2018 klo 12.45:

> Hi,
>
> Is there any similarities between these two requests in the JIRA regarding
> setting of prefer replica function?
>
> (SOLR-11982) Add support for preferReplicaTypes parameter
>
> (SOLR-8146) Allowing SolrJ CloudSolrClient to have preferred replica for
> query/read
>
> I am looking at setting one of the replica to be the preferred replica for
> query/read, and another replica to be use for indexing.
>
> I am using Solr 7.3.1 currently.
>
> Regards,
> Edwin
>

--
Ere Maijala
Kansalliskirjasto / The National Library of Finland
Reply | Threaded
Open this post in threaded view
|

Re: Setting preferred replica for query/read

Zheng Lin Edwin Yeo
Hi,

SOLR-8146 has not been updated since January last year, but I have just
commented it.

So we need both to be updated in order to achieve the full functionality of
setting preferred replica for query/read? Currently, is there a way to
achieve this by other means?

Regards,
Edwin

On 4 June 2018 at 19:43, Ere Maijala <[hidden email]> wrote:

> Hi,
>
> Well, SOLR-11982 adds server-side support for part of what SOLR-8146 aims
> to do (shards.preference=replica.location:[something]). It doesn't do
> regular expressions or snitches at the moment, though it would be easy to
> add. So, it looks to me like SOLR-8146 would need to be updated in this
> regard.
>
> --Ere
>
>
> Zheng Lin Edwin Yeo kirjoitti 4.6.2018 klo 12.45:
>
>> Hi,
>>
>> Is there any similarities between these two requests in the JIRA regarding
>> setting of prefer replica function?
>>
>> (SOLR-11982) Add support for preferReplicaTypes parameter
>>
>> (SOLR-8146) Allowing SolrJ CloudSolrClient to have preferred replica for
>> query/read
>>
>> I am looking at setting one of the replica to be the preferred replica for
>> query/read, and another replica to be use for indexing.
>>
>> I am using Solr 7.3.1 currently.
>>
>> Regards,
>> Edwin
>>
>>
> --
> Ere Maijala
> Kansalliskirjasto / The National Library of Finland
>
Reply | Threaded
Open this post in threaded view
|

Re: Setting preferred replica for query/read

Ere Maijala
Hi,

What I did in SOLR-11982 was meant to be used with replica types. The
idea is that you could have a set of NRT replicas used for indexing and
a set of PULL replicas used for queries. That's the easiest way to split
the work since PULL replicas never do indexing work, and then you can
say in the queries that "shards.preference=replica.type:PULL" or have
that as a default parameter in solrconfig. SOLR-8146 is not needed for
this. I suppose now that SOLR-11982 is done, SOLR-8146 would only be
needed to make it easier to set the preferred replica type etc.

SOLR-11982 also allows you to use replica location in node preference.
The nodes to use could be deduced from the cluster state and then you
could use shards.preference with replica.location. But that means the
client has to know which replicas to prefer.

Regards,
Ere

Zheng Lin Edwin Yeo kirjoitti 4.6.2018 klo 19.09:

> Hi,
>
> SOLR-8146 has not been updated since January last year, but I have just
> commented it.
>
> So we need both to be updated in order to achieve the full functionality of
> setting preferred replica for query/read? Currently, is there a way to
> achieve this by other means?
>
> Regards,
> Edwin
>
> On 4 June 2018 at 19:43, Ere Maijala <[hidden email]> wrote:
>
>> Hi,
>>
>> Well, SOLR-11982 adds server-side support for part of what SOLR-8146 aims
>> to do (shards.preference=replica.location:[something]). It doesn't do
>> regular expressions or snitches at the moment, though it would be easy to
>> add. So, it looks to me like SOLR-8146 would need to be updated in this
>> regard.
>>
>> --Ere
>>
>>
>> Zheng Lin Edwin Yeo kirjoitti 4.6.2018 klo 12.45:
>>
>>> Hi,
>>>
>>> Is there any similarities between these two requests in the JIRA regarding
>>> setting of prefer replica function?
>>>
>>> (SOLR-11982) Add support for preferReplicaTypes parameter
>>>
>>> (SOLR-8146) Allowing SolrJ CloudSolrClient to have preferred replica for
>>> query/read
>>>
>>> I am looking at setting one of the replica to be the preferred replica for
>>> query/read, and another replica to be use for indexing.
>>>
>>> I am using Solr 7.3.1 currently.
>>>
>>> Regards,
>>> Edwin
>>>
>>>
>> --
>> Ere Maijala
>> Kansalliskirjasto / The National Library of Finland
>>
>

--
Ere Maijala
Kansalliskirjasto / The National Library of Finland
Reply | Threaded
Open this post in threaded view
|

Re: Setting preferred replica for query/read

Zheng Lin Edwin Yeo
Hi Ere,

Thanks for your reply.

As currently we are looking at having a replica to do indexing, and another
replica to be use for searching, these 2 requests looks like it can archive
this purpose.

Will this be implemented in the Solr 7.4 release?

Regards,
Edwin


On 7 June 2018 at 16:00, Ere Maijala <[hidden email]> wrote:

> Hi,
>
> What I did in SOLR-11982 was meant to be used with replica types. The idea
> is that you could have a set of NRT replicas used for indexing and a set of
> PULL replicas used for queries. That's the easiest way to split the work
> since PULL replicas never do indexing work, and then you can say in the
> queries that "shards.preference=replica.type:PULL" or have that as a
> default parameter in solrconfig. SOLR-8146 is not needed for this. I
> suppose now that SOLR-11982 is done, SOLR-8146 would only be needed to make
> it easier to set the preferred replica type etc.
>
> SOLR-11982 also allows you to use replica location in node preference. The
> nodes to use could be deduced from the cluster state and then you could use
> shards.preference with replica.location. But that means the client has to
> know which replicas to prefer.
>
> Regards,
> Ere
>
>
> Zheng Lin Edwin Yeo kirjoitti 4.6.2018 klo 19.09:
>
>> Hi,
>>
>> SOLR-8146 has not been updated since January last year, but I have just
>> commented it.
>>
>> So we need both to be updated in order to achieve the full functionality
>> of
>> setting preferred replica for query/read? Currently, is there a way to
>> achieve this by other means?
>>
>> Regards,
>> Edwin
>>
>> On 4 June 2018 at 19:43, Ere Maijala <[hidden email]> wrote:
>>
>> Hi,
>>>
>>> Well, SOLR-11982 adds server-side support for part of what SOLR-8146 aims
>>> to do (shards.preference=replica.location:[something]). It doesn't do
>>> regular expressions or snitches at the moment, though it would be easy to
>>> add. So, it looks to me like SOLR-8146 would need to be updated in this
>>> regard.
>>>
>>> --Ere
>>>
>>>
>>> Zheng Lin Edwin Yeo kirjoitti 4.6.2018 klo 12.45:
>>>
>>> Hi,
>>>>
>>>> Is there any similarities between these two requests in the JIRA
>>>> regarding
>>>> setting of prefer replica function?
>>>>
>>>> (SOLR-11982) Add support for preferReplicaTypes parameter
>>>>
>>>> (SOLR-8146) Allowing SolrJ CloudSolrClient to have preferred replica for
>>>> query/read
>>>>
>>>> I am looking at setting one of the replica to be the preferred replica
>>>> for
>>>> query/read, and another replica to be use for indexing.
>>>>
>>>> I am using Solr 7.3.1 currently.
>>>>
>>>> Regards,
>>>> Edwin
>>>>
>>>>
>>>> --
>>> Ere Maijala
>>> Kansalliskirjasto / The National Library of Finland
>>>
>>>
>>
> --
> Ere Maijala
> Kansalliskirjasto / The National Library of Finland
>
Reply | Threaded
Open this post in threaded view
|

Re: Setting preferred replica for query/read

Shawn Heisey-2
On 6/7/2018 9:17 PM, Zheng Lin Edwin Yeo wrote:
> Thanks for your reply.
>
> As currently we are looking at having a replica to do indexing, and another
> replica to be use for searching, these 2 requests looks like it can archive
> this purpose.
>
> Will this be implemented in the Solr 7.4 release?

SOLR-11982 will be in version 7.4.0.

Having redundancy for indexing requires either two NRT replicas or two
TLOG replicas per shard.  You could do one of each, but I think it's
better to have them both the same type.

Then you can set up one PULL replica (or more if you wish), and use the
SOLR-11982 feature to indicate that you want to prefer PULL replicas.
You won't lose redundancy if there's only one PULL replica and it goes
down. In that situation, the NRT or TLOG replicas will be used.

Thanks,
Shawn

Reply | Threaded
Open this post in threaded view
|

Re: Setting preferred replica for query/read

Zheng Lin Edwin Yeo
Hi Shawn,

Just to confirm my understanding, if I have 2 replicas, I should set both
of them to either NRT replicas or TLOG replicas, and not one of each.
Then I set one of them to be PULL replica, which will be used for searching?

Regards,
Edwin

On 8 June 2018 at 12:54, Shawn Heisey <[hidden email]> wrote:

> On 6/7/2018 9:17 PM, Zheng Lin Edwin Yeo wrote:
>
>> Thanks for your reply.
>>
>> As currently we are looking at having a replica to do indexing, and
>> another
>> replica to be use for searching, these 2 requests looks like it can
>> archive
>> this purpose.
>>
>> Will this be implemented in the Solr 7.4 release?
>>
>
> SOLR-11982 will be in version 7.4.0.
>
> Having redundancy for indexing requires either two NRT replicas or two
> TLOG replicas per shard.  You could do one of each, but I think it's better
> to have them both the same type.
>
> Then you can set up one PULL replica (or more if you wish), and use the
> SOLR-11982 feature to indicate that you want to prefer PULL replicas. You
> won't lose redundancy if there's only one PULL replica and it goes down. In
> that situation, the NRT or TLOG replicas will be used.
>
> Thanks,
> Shawn
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Setting preferred replica for query/read

Shawn Heisey-2
On 6/9/2018 8:14 AM, Zheng Lin Edwin Yeo wrote:
> Just to confirm my understanding, if I have 2 replicas, I should set both
> of them to either NRT replicas or TLOG replicas, and not one of each.
> Then I set one of them to be PULL replica, which will be used for searching?

There are multiple possible combinations that will work. My opinion is
that one of these combinations should be used:

Combo 1:
2 NRT replicas.
1 or more PULL replicas.
Set preferred to PULL with SOLR-11982 functionality.

Combo 2:
2 TLOG replicas.
1 or more PULL replicas.
Set preferred to PULL with SOLR-11982 functionality.

With either of those combos, behavior will be identical no matter which
replica is leader.  The second option would have less work to do, but
I'm not sure that it would actually have any better performance than the
first option.

If you instead choose to have 1 NRT, 1 TLOG, and 1 or more PULL, then
cluster behavior would be different when the NRT replica is leader than
it is when the TLOG replica is leader.It would *work*, though ... and
you might not even notice a difference unless you look closely into how
it's behaving.

For clarity, keep in mind that a PULL replica cannot become leader.

Also keep in mind that change visibility with soft commits is only
guaranteed when EVERY replica is NRT.  This is mentioned in the
documentation for 7.x versions.

https://lucene.apache.org/solr/guide/7_3/shards-and-indexing-data-in-solrcloud.html#combining-replica-types-in-a-cluster

So if you choose to prefer pull replicas, you will either have to switch
to hard commits for visibility, or live with search results sometimes
not being completely current.

Thanks,
Shawn

Reply | Threaded
Open this post in threaded view
|

Re: Setting preferred replica for query/read

Zheng Lin Edwin Yeo
Hi Shawn,

Thanks for the info.
Looking forward to having this functionality in Solr 7.4.0.

Regards,
Edwin

On 9 June 2018 at 23:02, Shawn Heisey <[hidden email]> wrote:

> On 6/9/2018 8:14 AM, Zheng Lin Edwin Yeo wrote:
>
>> Just to confirm my understanding, if I have 2 replicas, I should set both
>> of them to either NRT replicas or TLOG replicas, and not one of each.
>> Then I set one of them to be PULL replica, which will be used for
>> searching?
>>
>
> There are multiple possible combinations that will work. My opinion is
> that one of these combinations should be used:
>
> Combo 1:
> 2 NRT replicas.
> 1 or more PULL replicas.
> Set preferred to PULL with SOLR-11982 functionality.
>
> Combo 2:
> 2 TLOG replicas.
> 1 or more PULL replicas.
> Set preferred to PULL with SOLR-11982 functionality.
>
> With either of those combos, behavior will be identical no matter which
> replica is leader.  The second option would have less work to do, but I'm
> not sure that it would actually have any better performance than the first
> option.
>
> If you instead choose to have 1 NRT, 1 TLOG, and 1 or more PULL, then
> cluster behavior would be different when the NRT replica is leader than it
> is when the TLOG replica is leader.It would *work*, though ... and you
> might not even notice a difference unless you look closely into how it's
> behaving.
>
> For clarity, keep in mind that a PULL replica cannot become leader.
>
> Also keep in mind that change visibility with soft commits is only
> guaranteed when EVERY replica is NRT.  This is mentioned in the
> documentation for 7.x versions.
>
> https://lucene.apache.org/solr/guide/7_3/shards-and-indexing
> -data-in-solrcloud.html#combining-replica-types-in-a-cluster
>
> So if you choose to prefer pull replicas, you will either have to switch
> to hard commits for visibility, or live with search results sometimes not
> being completely current.
>
> Thanks,
> Shawn
>
>