Rule-based replication or sharding

classic Classic list List threaded Threaded
22 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Rule-based replication or sharding

Chuck Reynolds
This post was updated on .
I'm using Solr 6.6 and I want to create a 90 node cluster with a replication
factor of three.  I'm using AWS EC2 instances and I have a requirement to
replicate the data into 3 AWS availability zones.  

So 30 servers in each zone and I don't see a create collection rule that
will put one replica in each of the three zones.

What am I missing?



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Reply | Threaded
Open this post in threaded view
|

Re: Rule-based replication or sharing

sarowe
Hi Chuck,

One way to do it is to set a system property on the JVM running each Solr node, corresponding to the the AWS availability zone on which the node is hosted.

For example, you could use sysprop “AWSAZ”, then use rules like:

   replica:<2,sysprop.AWSAZ:us-east-1
   replica:<2,sysprop.AWSAZ:us-west-1
   replica:<2,sysprop.AWSAZ:ca-central-1

--
Steve
www.lucidworks.com

> On Sep 21, 2018, at 4:07 PM, Chuck Reynolds <[hidden email]> wrote:
>
> I'm using Solr 6.6 and I want to create a 90 node cluster with a replication
> factor of three.  I'm using AWS EC2 instances and I have a requirement to
> replicate the data into 3 AWS availability zones.  
>
> So 30 servers in each zone and I don't see a create collection rule that
> will put one replica in each of the three zones.
>
> What am I missing?
>
>
>
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Reply | Threaded
Open this post in threaded view
|

Re: Rule-based replication or sharing

Shawn Heisey-2
In reply to this post by Chuck Reynolds
On 9/21/2018 2:07 PM, Chuck Reynolds wrote:
> I'm using Solr 6.6 and I want to create a 90 node cluster with a replication
> factor of three.  I'm using AWS EC2 instances and I have a requirement to
> replicate the data into 3 AWS availability zones.
>
> So 30 servers in each zone and I don't see a create collection rule that
> will put one replica in each of the three zones.
>
> What am I missing?

The documentation says that one of the uses is to do things like only
allowing one replica per rack.

http://lucene.apache.org/solr/guide/7_4/rule-based-replica-placement.html

But it doesn't say how to do this.  Reading the documentation, I cannot
figure it out.

There's a little more data on the issue that implemented the feature:

https://jira.apache.org/jira/browse/SOLR-6220

But I still can't tell from reading both of these how to actually DO it.

Unanswered questions:

How do I assign arbitrary tags to each node that are then used in a snitch?
How are these rules added to Solr's configuration?
The MODIFYCOLLECTION action is mentioned.  But what if I want to create
rules for collections that don't exist yet?  I can't modify a collection
if it doesn't exist yet.
Exactly what text is required, and where?  SOLR-6220 shows JSON-type
syntax, but that type of syntax is not mentioned in the documentation AT
ALL.

I remember seeing something come through somewhere (jira? dev list? not
sure where!) about this feature being deprecated in favor of autoscale
settings, but I can't find it now.  If autoscaling is the preferred way
to handle this, how is it done? I can't find anything useful in that
documentation section either.

Thanks,
Shawn

Reply | Threaded
Open this post in threaded view
|

Re: Rule-based replication or sharing

Chuck Reynolds
In reply to this post by sarowe
Thanks Steve,

I saw this option but it will mean a bit of re-working our automation to implement.

The documentation talks about an EC2Snitch. I wish it worked like Cassandra where you just say that's the one you're using and it figures out how to replicate the data

On 9/21/18, 2:40 PM, "Steve Rowe" <[hidden email]> wrote:

    Hi Chuck,
   
    One way to do it is to set a system property on the JVM running each Solr node, corresponding to the the AWS availability zone on which the node is hosted.
   
    For example, you could use sysprop “AWSAZ”, then use rules like:
   
       replica:<2,sysprop.AWSAZ:us-east-1
       replica:<2,sysprop.AWSAZ:us-west-1
       replica:<2,sysprop.AWSAZ:ca-central-1
   
    --
    Steve
    https://urldefense.proofpoint.com/v2/url?u=http-3A__www.lucidworks.com&d=DwIFaQ&c=kKqjBR9KKWaWpMhASkPbOg&r=J-2s3b-3-OTA0o6bGDhJXAQlB5Y3s4rOUxlh_78DJl0&m=glt-Kw4TwOAGYMt6NB7R6qMysuNssE_CjJH46rL4tqo&s=6CzANqo-EwE1nnzHaTwr71MxQd7-im366kZUXznMKC8&e=
   
    > On Sep 21, 2018, at 4:07 PM, Chuck Reynolds <[hidden email]> wrote:
    >
    > I'm using Solr 6.6 and I want to create a 90 node cluster with a replication
    > factor of three.  I'm using AWS EC2 instances and I have a requirement to
    > replicate the data into 3 AWS availability zones.  
    >
    > So 30 servers in each zone and I don't see a create collection rule that
    > will put one replica in each of the three zones.
    >
    > What am I missing?
    >
    >
    >
    > --
    > Sent from: https://urldefense.proofpoint.com/v2/url?u=http-3A__lucene.472066.n3.nabble.com_Solr-2DUser-2Df472068.html&d=DwIFaQ&c=kKqjBR9KKWaWpMhASkPbOg&r=J-2s3b-3-OTA0o6bGDhJXAQlB5Y3s4rOUxlh_78DJl0&m=glt-Kw4TwOAGYMt6NB7R6qMysuNssE_CjJH46rL4tqo&s=pnPq-r9xSpo7DZsgF-XgR0MyUIFNcaZpAI-xcX4HjCY&e=
   
   

Reply | Threaded
Open this post in threaded view
|

Re: Rule-based replication or sharing

Chuck Reynolds
In reply to this post by sarowe
Steve,

I wasn't able to get the sysprop to work.  I think maybe there is a disconnect on my part.

From the documentation it looks like I can only use the sysprop tag if I'm using a Snitch.  Is that correct.

I can't find any example of anyone using the default Snitch.

Here is what I have for my rule:
rule=shard:*,replica:1,sysprop.AWSAZ:AZ1&rule=shard:*,replica:2,sysprop.AWSAZ:AZ2&rule=shard:*,replica:3,sysprop.AWSAZ:AZ3

I'm not specifying a snitch.  Is that my problem or is there a problem with my rule?

Thanks for your help.
On 9/21/18, 2:40 PM, "Steve Rowe" <[hidden email]> wrote:

    Hi Chuck,
   
    One way to do it is to set a system property on the JVM running each Solr node, corresponding to the the AWS availability zone on which the node is hosted.
   
    For example, you could use sysprop “AWSAZ”, then use rules like:
   
       replica:<2,sysprop.AWSAZ:us-east-1
       replica:<2,sysprop.AWSAZ:us-west-1
       replica:<2,sysprop.AWSAZ:ca-central-1
   
    --
    Steve
    https://urldefense.proofpoint.com/v2/url?u=http-3A__www.lucidworks.com&d=DwIFaQ&c=kKqjBR9KKWaWpMhASkPbOg&r=J-2s3b-3-OTA0o6bGDhJXAQlB5Y3s4rOUxlh_78DJl0&m=glt-Kw4TwOAGYMt6NB7R6qMysuNssE_CjJH46rL4tqo&s=6CzANqo-EwE1nnzHaTwr71MxQd7-im366kZUXznMKC8&e=
   
    > On Sep 21, 2018, at 4:07 PM, Chuck Reynolds <[hidden email]> wrote:
    >
    > I'm using Solr 6.6 and I want to create a 90 node cluster with a replication
    > factor of three.  I'm using AWS EC2 instances and I have a requirement to
    > replicate the data into 3 AWS availability zones.  
    >
    > So 30 servers in each zone and I don't see a create collection rule that
    > will put one replica in each of the three zones.
    >
    > What am I missing?
    >
    >
    >
    > --
    > Sent from: https://urldefense.proofpoint.com/v2/url?u=http-3A__lucene.472066.n3.nabble.com_Solr-2DUser-2Df472068.html&d=DwIFaQ&c=kKqjBR9KKWaWpMhASkPbOg&r=J-2s3b-3-OTA0o6bGDhJXAQlB5Y3s4rOUxlh_78DJl0&m=glt-Kw4TwOAGYMt6NB7R6qMysuNssE_CjJH46rL4tqo&s=pnPq-r9xSpo7DZsgF-XgR0MyUIFNcaZpAI-xcX4HjCY&e=
   
   

Reply | Threaded
Open this post in threaded view
|

Re: Rule-based replication or sharing

sarowe
Chuck,

The default Snitch is the one that’s used if you don’t specify one in a rule.  The sysprop.* tag is provided by the default Snitch.

The only thing that seems wrong to me in your rules is “replica:1”, “replica:2”, and “replica:3” - these say that exactly one, two, and three replicas of each shard, respectively, must be on each of the nodes that has the respective sysprop value.

Since these rules will apply to all nodes that match the sysprop value, you have to allow for the possibility that some nodes will have *zero* replicas of a shard.  So you could specify “replica:<2”, which means that no node can host more than one replica, but it's acceptable for a node to host zero replicas.

Did you set system property AWSAZ on each Solr node with an appropriate value?

--
Steve
www.lucidworks.com

> On Sep 25, 2018, at 10:39 AM, Chuck Reynolds <[hidden email]> wrote:
>
> Steve,
>
> I wasn't able to get the sysprop to work.  I think maybe there is a disconnect on my part.
>
> From the documentation it looks like I can only use the sysprop tag if I'm using a Snitch.  Is that correct.
>
> I can't find any example of anyone using the default Snitch.
>
> Here is what I have for my rule:
> rule=shard:*,replica:1,sysprop.AWSAZ:AZ1&rule=shard:*,replica:2,sysprop.AWSAZ:AZ2&rule=shard:*,replica:3,sysprop.AWSAZ:AZ3
>
> I'm not specifying a snitch.  Is that my problem or is there a problem with my rule?
>
> Thanks for your help.
> On 9/21/18, 2:40 PM, "Steve Rowe" <[hidden email]> wrote:
>
>    Hi Chuck,
>
>    One way to do it is to set a system property on the JVM running each Solr node, corresponding to the the AWS availability zone on which the node is hosted.
>
>    For example, you could use sysprop “AWSAZ”, then use rules like:
>
>       replica:<2,sysprop.AWSAZ:us-east-1
>       replica:<2,sysprop.AWSAZ:us-west-1
>       replica:<2,sysprop.AWSAZ:ca-central-1
>
>    --
>    Steve
>    https://urldefense.proofpoint.com/v2/url?u=http-3A__www.lucidworks.com&d=DwIFaQ&c=kKqjBR9KKWaWpMhASkPbOg&r=J-2s3b-3-OTA0o6bGDhJXAQlB5Y3s4rOUxlh_78DJl0&m=glt-Kw4TwOAGYMt6NB7R6qMysuNssE_CjJH46rL4tqo&s=6CzANqo-EwE1nnzHaTwr71MxQd7-im366kZUXznMKC8&e=
>
>> On Sep 21, 2018, at 4:07 PM, Chuck Reynolds <[hidden email]> wrote:
>>
>> I'm using Solr 6.6 and I want to create a 90 node cluster with a replication
>> factor of three.  I'm using AWS EC2 instances and I have a requirement to
>> replicate the data into 3 AWS availability zones.  
>>
>> So 30 servers in each zone and I don't see a create collection rule that
>> will put one replica in each of the three zones.
>>
>> What am I missing?
>>
>>
>>
>> --
>> Sent from: https://urldefense.proofpoint.com/v2/url?u=http-3A__lucene.472066.n3.nabble.com_Solr-2DUser-2Df472068.html&d=DwIFaQ&c=kKqjBR9KKWaWpMhASkPbOg&r=J-2s3b-3-OTA0o6bGDhJXAQlB5Y3s4rOUxlh_78DJl0&m=glt-Kw4TwOAGYMt6NB7R6qMysuNssE_CjJH46rL4tqo&s=pnPq-r9xSpo7DZsgF-XgR0MyUIFNcaZpAI-xcX4HjCY&e=
>
>
>

Reply | Threaded
Open this post in threaded view
|

Re: Rule-based replication or sharing

Chuck Reynolds
Steve,

Yes I set the set system property AWSAZ and I've checked the java properties in Solr and I can see them.

It maybe the way we are configuring Solr so let me explain that first.

So we have 90 server in AWS, 30 servers per AZ's.
90 shards for the cluster.
Each server has three instances of Solr running on it so every instance on the server has to be in the same replica set.
So for example shard 1 will have three replicas and each replica needs to be in a separate AZ.

So does the rule of replica:>2 work?

I just need all of the servers in an AZ to be in the same replica.  Does that make sense?

On 9/25/18, 9:08 AM, "Steve Rowe" <[hidden email]> wrote:

    Chuck,
   
    The default Snitch is the one that’s used if you don’t specify one in a rule.  The sysprop.* tag is provided by the default Snitch.
   
    The only thing that seems wrong to me in your rules is “replica:1”, “replica:2”, and “replica:3” - these say that exactly one, two, and three replicas of each shard, respectively, must be on each of the nodes that has the respective sysprop value.
   
    Since these rules will apply to all nodes that match the sysprop value, you have to allow for the possibility that some nodes will have *zero* replicas of a shard.  So you could specify “replica:<2”, which means that no node can host more than one replica, but it's acceptable for a node to host zero replicas.
   
    Did you set system property AWSAZ on each Solr node with an appropriate value?
   
    --
    Steve
    https://urldefense.proofpoint.com/v2/url?u=http-3A__www.lucidworks.com&d=DwIFaQ&c=kKqjBR9KKWaWpMhASkPbOg&r=J-2s3b-3-OTA0o6bGDhJXAQlB5Y3s4rOUxlh_78DJl0&m=uG91WrgZB5UTKLOAB53AcrY5LyBsJ3VyBH8cN7xe2mU&s=9V6TJoE0h5NMjEWZ38ipa3zgYvoLJ1H9GHplSz1DJLU&e=
   
    > On Sep 25, 2018, at 10:39 AM, Chuck Reynolds <[hidden email]> wrote:
    >
    > Steve,
    >
    > I wasn't able to get the sysprop to work.  I think maybe there is a disconnect on my part.
    >
    > From the documentation it looks like I can only use the sysprop tag if I'm using a Snitch.  Is that correct.
    >
    > I can't find any example of anyone using the default Snitch.
    >
    > Here is what I have for my rule:
    > rule=shard:*,replica:1,sysprop.AWSAZ:AZ1&rule=shard:*,replica:2,sysprop.AWSAZ:AZ2&rule=shard:*,replica:3,sysprop.AWSAZ:AZ3
    >
    > I'm not specifying a snitch.  Is that my problem or is there a problem with my rule?
    >
    > Thanks for your help.
    > On 9/21/18, 2:40 PM, "Steve Rowe" <[hidden email]> wrote:
    >
    >    Hi Chuck,
    >
    >    One way to do it is to set a system property on the JVM running each Solr node, corresponding to the the AWS availability zone on which the node is hosted.
    >
    >    For example, you could use sysprop “AWSAZ”, then use rules like:
    >
    >       replica:<2,sysprop.AWSAZ:us-east-1
    >       replica:<2,sysprop.AWSAZ:us-west-1
    >       replica:<2,sysprop.AWSAZ:ca-central-1
    >
    >    --
    >    Steve
    >    https://urldefense.proofpoint.com/v2/url?u=http-3A__www.lucidworks.com&d=DwIFaQ&c=kKqjBR9KKWaWpMhASkPbOg&r=J-2s3b-3-OTA0o6bGDhJXAQlB5Y3s4rOUxlh_78DJl0&m=glt-Kw4TwOAGYMt6NB7R6qMysuNssE_CjJH46rL4tqo&s=6CzANqo-EwE1nnzHaTwr71MxQd7-im366kZUXznMKC8&e=
    >
    >> On Sep 21, 2018, at 4:07 PM, Chuck Reynolds <[hidden email]> wrote:
    >>
    >> I'm using Solr 6.6 and I want to create a 90 node cluster with a replication
    >> factor of three.  I'm using AWS EC2 instances and I have a requirement to
    >> replicate the data into 3 AWS availability zones.  
    >>
    >> So 30 servers in each zone and I don't see a create collection rule that
    >> will put one replica in each of the three zones.
    >>
    >> What am I missing?
    >>
    >>
    >>
    >> --
    >> Sent from: https://urldefense.proofpoint.com/v2/url?u=http-3A__lucene.472066.n3.nabble.com_Solr-2DUser-2Df472068.html&d=DwIFaQ&c=kKqjBR9KKWaWpMhASkPbOg&r=J-2s3b-3-OTA0o6bGDhJXAQlB5Y3s4rOUxlh_78DJl0&m=glt-Kw4TwOAGYMt6NB7R6qMysuNssE_CjJH46rL4tqo&s=pnPq-r9xSpo7DZsgF-XgR0MyUIFNcaZpAI-xcX4HjCY&e=
    >
    >
    >
   
   

Reply | Threaded
Open this post in threaded view
|

Re: Rule-based replication or sharing

sarowe
Hi Chuck, see my replies inline below:

> On Sep 25, 2018, at 11:21 AM, Chuck Reynolds <[hidden email]> wrote:
>
> So we have 90 server in AWS, 30 servers per AZ's.
> 90 shards for the cluster.
> Each server has three instances of Solr running on it so every instance on the server has to be in the same replica set.

You lost me here.  What is a "replica set”?  And why does each instance of Solr (referred to in the reference guide as a “node”, BTW) running on a server need to be in the same “replica set”?

(I’m guessing you have theorized that “replica:3” is a way of referring to "replica set #3”, but that’s incorrect; “replica:3” means that exactly 3 replicas must be placed on the bucket of nodes you specify in the rule; more info below.)

> So for example shard 1 will have three replicas and each replica needs to be in a separate AZ.

Okay, I understand this part, but I fail to see how this is an example of your “replica set” assertion above.

> So does the rule of replica:>2 work?

I assume you did not mean ^^ literally, since you wrote “>” where I wrote “<“ in my previous response.

I checked offline with Noble Paul, who wrote the rule-based replica placement feature, and he corrected a misunderstanding of mine:

> On 9/25/18, 9:08 AM, "Steve Rowe" <[hidden email]> wrote:

> So you could specify “replica:<2”, which means that no node can host more than one replica, but it's acceptable for a node to host zero replicas.

But ^^ is incorrect.

“replica:<2” means that either zero or one replica of each shard of the collection to be created may be hosted on the bucket of *all* of the nodes that have the specified AWSAZ sysprop value.  That is, when placing replicas, Solr will put either zero or one replica on one of the nodes in the bucket.  And AFAICT that’s not exatly what you want, since zero replicas of a shard on an AZ is not acceptable.

> I just need all of the servers in an AZ to be in the same replica.  Does that make sense?

I’m not sure?  This sounds like something different from your above example: "shard 1 will have three replicas and each replica needs to be in a separate AZ.”

If you mean “exactly one Solr instance in an AZ must host exactly one replica of each shard of the collection”, then yes, that makes sense :).

Okay, one more try :) - here are the rules that should do the trick for you (i.e., what I wrote in the previous sentence):

-----
 rule=shard:*,replica:1,sysprop.AWSAZ:AZ1
&rule=shard:*,replica:1,sysprop.AWSAZ:AZ2
&rule=shard:*,replica:1,sysprop.AWSAZ:AZ3
-----

--
Steve
www.lucidworks.com

> On 9/25/18, 9:08 AM, "Steve Rowe" <[hidden email]> wrote:
>
>    Chuck,
>
>    The default Snitch is the one that’s used if you don’t specify one in a rule.  The sysprop.* tag is provided by the default Snitch.
>
>    The only thing that seems wrong to me in your rules is “replica:1”, “replica:2”, and “replica:3” - these say that exactly one, two, and three replicas of each shard, respectively, must be on each of the nodes that has the respective sysprop value.
>
>    Since these rules will apply to all nodes that match the sysprop value, you have to allow for the possibility that some nodes will have *zero* replicas of a shard.  So you could specify “replica:<2”, which means that no node can host more than one replica, but it's acceptable for a node to host zero replicas.
>
>    Did you set system property AWSAZ on each Solr node with an appropriate value?
>
>    --
>    Steve
>    https://urldefense.proofpoint.com/v2/url?u=http-3A__www.lucidworks.com&d=DwIFaQ&c=kKqjBR9KKWaWpMhASkPbOg&r=J-2s3b-3-OTA0o6bGDhJXAQlB5Y3s4rOUxlh_78DJl0&m=uG91WrgZB5UTKLOAB53AcrY5LyBsJ3VyBH8cN7xe2mU&s=9V6TJoE0h5NMjEWZ38ipa3zgYvoLJ1H9GHplSz1DJLU&e=
>
>> On Sep 25, 2018, at 10:39 AM, Chuck Reynolds <[hidden email]> wrote:
>>
>> Steve,
>>
>> I wasn't able to get the sysprop to work.  I think maybe there is a disconnect on my part.
>>
>> From the documentation it looks like I can only use the sysprop tag if I'm using a Snitch.  Is that correct.
>>
>> I can't find any example of anyone using the default Snitch.
>>
>> Here is what I have for my rule:
>> rule=shard:*,replica:1,sysprop.AWSAZ:AZ1&rule=shard:*,replica:2,sysprop.AWSAZ:AZ2&rule=shard:*,replica:3,sysprop.AWSAZ:AZ3
>>
>> I'm not specifying a snitch.  Is that my problem or is there a problem with my rule?
>>
>> Thanks for your help.
>> On 9/21/18, 2:40 PM, "Steve Rowe" <[hidden email]> wrote:
>>
>>   Hi Chuck,
>>
>>   One way to do it is to set a system property on the JVM running each Solr node, corresponding to the the AWS availability zone on which the node is hosted.
>>
>>   For example, you could use sysprop “AWSAZ”, then use rules like:
>>
>>      replica:<2,sysprop.AWSAZ:us-east-1
>>      replica:<2,sysprop.AWSAZ:us-west-1
>>      replica:<2,sysprop.AWSAZ:ca-central-1
>>
>>   --
>>   Steve
>>   https://urldefense.proofpoint.com/v2/url?u=http-3A__www.lucidworks.com&d=DwIFaQ&c=kKqjBR9KKWaWpMhASkPbOg&r=J-2s3b-3-OTA0o6bGDhJXAQlB5Y3s4rOUxlh_78DJl0&m=glt-Kw4TwOAGYMt6NB7R6qMysuNssE_CjJH46rL4tqo&s=6CzANqo-EwE1nnzHaTwr71MxQd7-im366kZUXznMKC8&e=
>>
>>> On Sep 21, 2018, at 4:07 PM, Chuck Reynolds <[hidden email]> wrote:
>>>
>>> I'm using Solr 6.6 and I want to create a 90 node cluster with a replication
>>> factor of three.  I'm using AWS EC2 instances and I have a requirement to
>>> replicate the data into 3 AWS availability zones.  
>>>
>>> So 30 servers in each zone and I don't see a create collection rule that
>>> will put one replica in each of the three zones.
>>>
>>> What am I missing?
>>>
>>>
>>>
>>> --
>>> Sent from: https://urldefense.proofpoint.com/v2/url?u=http-3A__lucene.472066.n3.nabble.com_Solr-2DUser-2Df472068.html&d=DwIFaQ&c=kKqjBR9KKWaWpMhASkPbOg&r=J-2s3b-3-OTA0o6bGDhJXAQlB5Y3s4rOUxlh_78DJl0&m=glt-Kw4TwOAGYMt6NB7R6qMysuNssE_CjJH46rL4tqo&s=pnPq-r9xSpo7DZsgF-XgR0MyUIFNcaZpAI-xcX4HjCY&e=

Reply | Threaded
Open this post in threaded view
|

Re: Rule-based replication or sharing

Chuck Reynolds
Steve,

No doubt I confused you.  I'm confused myself__

When I said replica set what I was referring to was one of the three replicas of the data.  Each replica needing to be in a different AZ.

What is a "replica set”?  And why does each instance of Solr (referred to in the reference guide as a “node”, BTW) running on a server need to be in the same “replica set”?
        What I should of said is each node on the server which there are three per server needs to be in the same AZ.

The bottom line is I guess I'm confused by the documentation and the reference to replicas. Normally when referring to replicas in the documentation it is referring to the number of times you want the data replicated. As in replication factor.  That's where the confusion was for me.

So let me ask this simple question.

If I want to create a rule that insures that my replication factor of three correctly shards the data across three AZ so if I was to lose one or even two AZ's in AWS Solr would still have 1 - 2 copies of the data.   How would that rule work?



On 9/25/18, 10:17 AM, "Steve Rowe" <[hidden email]> wrote:

    Hi Chuck, see my replies inline below:
   
    > On Sep 25, 2018, at 11:21 AM, Chuck Reynolds <[hidden email]> wrote:
    >
    > So we have 90 server in AWS, 30 servers per AZ's.
    > 90 shards for the cluster.
    > Each server has three instances of Solr running on it so every instance on the server has to be in the same replica set.
   
    You lost me here.  What is a "replica set”?  And why does each instance of Solr (referred to in the reference guide as a “node”, BTW) running on a server need to be in the same “replica set”?
   
    (I’m guessing you have theorized that “replica:3” is a way of referring to "replica set #3”, but that’s incorrect; “replica:3” means that exactly 3 replicas must be placed on the bucket of nodes you specify in the rule; more info below.)
   
    > So for example shard 1 will have three replicas and each replica needs to be in a separate AZ.
   
    Okay, I understand this part, but I fail to see how this is an example of your “replica set” assertion above.
   
    > So does the rule of replica:>2 work?
   
    I assume you did not mean ^^ literally, since you wrote “>” where I wrote “<“ in my previous response.
   
    I checked offline with Noble Paul, who wrote the rule-based replica placement feature, and he corrected a misunderstanding of mine:
   
    > On 9/25/18, 9:08 AM, "Steve Rowe" <[hidden email]> wrote:
   
    > So you could specify “replica:<2”, which means that no node can host more than one replica, but it's acceptable for a node to host zero replicas.
   
    But ^^ is incorrect.
   
    “replica:<2” means that either zero or one replica of each shard of the collection to be created may be hosted on the bucket of *all* of the nodes that have the specified AWSAZ sysprop value.  That is, when placing replicas, Solr will put either zero or one replica on one of the nodes in the bucket.  And AFAICT that’s not exatly what you want, since zero replicas of a shard on an AZ is not acceptable.
   
    > I just need all of the servers in an AZ to be in the same replica.  Does that make sense?
   
    I’m not sure?  This sounds like something different from your above example: "shard 1 will have three replicas and each replica needs to be in a separate AZ.”
   
    If you mean “exactly one Solr instance in an AZ must host exactly one replica of each shard of the collection”, then yes, that makes sense :).
   
    Okay, one more try :) - here are the rules that should do the trick for you (i.e., what I wrote in the previous sentence):
   
    -----
     rule=shard:*,replica:1,sysprop.AWSAZ:AZ1
    &rule=shard:*,replica:1,sysprop.AWSAZ:AZ2
    &rule=shard:*,replica:1,sysprop.AWSAZ:AZ3
    -----
   
    --
    Steve
    https://urldefense.proofpoint.com/v2/url?u=http-3A__www.lucidworks.com&d=DwIFaQ&c=kKqjBR9KKWaWpMhASkPbOg&r=J-2s3b-3-OTA0o6bGDhJXAQlB5Y3s4rOUxlh_78DJl0&m=e6E6P07QRR0Gn9oI2V1tk3sWdVDq5EF_tIgdoh4DxpE&s=Ddb5KOc_t4p64xyxt5rmqnWWwMcByecGQ2iJYv2BWiY&e=
   
    > On 9/25/18, 9:08 AM, "Steve Rowe" <[hidden email]> wrote:
    >
    >    Chuck,
    >
    >    The default Snitch is the one that’s used if you don’t specify one in a rule.  The sysprop.* tag is provided by the default Snitch.
    >
    >    The only thing that seems wrong to me in your rules is “replica:1”, “replica:2”, and “replica:3” - these say that exactly one, two, and three replicas of each shard, respectively, must be on each of the nodes that has the respective sysprop value.
    >
    >    Since these rules will apply to all nodes that match the sysprop value, you have to allow for the possibility that some nodes will have *zero* replicas of a shard.  So you could specify “replica:<2”, which means that no node can host more than one replica, but it's acceptable for a node to host zero replicas.
    >
    >    Did you set system property AWSAZ on each Solr node with an appropriate value?
    >
    >    --
    >    Steve
    >    https://urldefense.proofpoint.com/v2/url?u=http-3A__www.lucidworks.com&d=DwIFaQ&c=kKqjBR9KKWaWpMhASkPbOg&r=J-2s3b-3-OTA0o6bGDhJXAQlB5Y3s4rOUxlh_78DJl0&m=uG91WrgZB5UTKLOAB53AcrY5LyBsJ3VyBH8cN7xe2mU&s=9V6TJoE0h5NMjEWZ38ipa3zgYvoLJ1H9GHplSz1DJLU&e=
    >
    >> On Sep 25, 2018, at 10:39 AM, Chuck Reynolds <[hidden email]> wrote:
    >>
    >> Steve,
    >>
    >> I wasn't able to get the sysprop to work.  I think maybe there is a disconnect on my part.
    >>
    >> From the documentation it looks like I can only use the sysprop tag if I'm using a Snitch.  Is that correct.
    >>
    >> I can't find any example of anyone using the default Snitch.
    >>
    >> Here is what I have for my rule:
    >> rule=shard:*,replica:1,sysprop.AWSAZ:AZ1&rule=shard:*,replica:2,sysprop.AWSAZ:AZ2&rule=shard:*,replica:3,sysprop.AWSAZ:AZ3
    >>
    >> I'm not specifying a snitch.  Is that my problem or is there a problem with my rule?
    >>
    >> Thanks for your help.
    >> On 9/21/18, 2:40 PM, "Steve Rowe" <[hidden email]> wrote:
    >>
    >>   Hi Chuck,
    >>
    >>   One way to do it is to set a system property on the JVM running each Solr node, corresponding to the the AWS availability zone on which the node is hosted.
    >>
    >>   For example, you could use sysprop “AWSAZ”, then use rules like:
    >>
    >>      replica:<2,sysprop.AWSAZ:us-east-1
    >>      replica:<2,sysprop.AWSAZ:us-west-1
    >>      replica:<2,sysprop.AWSAZ:ca-central-1
    >>
    >>   --
    >>   Steve
    >>   https://urldefense.proofpoint.com/v2/url?u=http-3A__www.lucidworks.com&d=DwIFaQ&c=kKqjBR9KKWaWpMhASkPbOg&r=J-2s3b-3-OTA0o6bGDhJXAQlB5Y3s4rOUxlh_78DJl0&m=glt-Kw4TwOAGYMt6NB7R6qMysuNssE_CjJH46rL4tqo&s=6CzANqo-EwE1nnzHaTwr71MxQd7-im366kZUXznMKC8&e=
    >>
    >>> On Sep 21, 2018, at 4:07 PM, Chuck Reynolds <[hidden email]> wrote:
    >>>
    >>> I'm using Solr 6.6 and I want to create a 90 node cluster with a replication
    >>> factor of three.  I'm using AWS EC2 instances and I have a requirement to
    >>> replicate the data into 3 AWS availability zones.  
    >>>
    >>> So 30 servers in each zone and I don't see a create collection rule that
    >>> will put one replica in each of the three zones.
    >>>
    >>> What am I missing?
    >>>
    >>>
    >>>
    >>> --
    >>> Sent from: https://urldefense.proofpoint.com/v2/url?u=http-3A__lucene.472066.n3.nabble.com_Solr-2DUser-2Df472068.html&d=DwIFaQ&c=kKqjBR9KKWaWpMhASkPbOg&r=J-2s3b-3-OTA0o6bGDhJXAQlB5Y3s4rOUxlh_78DJl0&m=glt-Kw4TwOAGYMt6NB7R6qMysuNssE_CjJH46rL4tqo&s=pnPq-r9xSpo7DZsgF-XgR0MyUIFNcaZpAI-xcX4HjCY&e=
   
   

Reply | Threaded
Open this post in threaded view
|

Re: Rule-based replication or sharing

sarowe
Chuck, see my responses inline below:

> On Sep 25, 2018, at 12:50 PM, Chuck Reynolds <[hidden email]> wrote:
> The bottom line is I guess I'm confused by the documentation and the reference to replicas. Normally when referring to replicas in the documentation it is referring to the number of times you want the data replicated. As in replication factor.  That's where the confusion was for me.

We can always use help improving Solr’s documentation, and your perspective is valuable.  Please see https://wiki.apache.org/solr/HowToContribute and open JIRA issues with the problems you find, and ideally with patches against the ref guide sources.

From the Solr 6.4 ref guide’s “Solr Glossary”:

  Replica: A Core that acts as a physical copy of a Shard in a SolrCloud Collection.

As ^^ indicates, a “replica” is not a replication factor.  (Though “replica:1” in a rule-based replica placement rule is a condition on replica *count*, so I can see where that could be confusing.)

> If I want to create a rule that insures that my replication factor of three correctly shards the data across three AZ so if I was to lose one or even two AZ's in AWS Solr would still have 1 - 2 copies of the data.   How would that rule work?

I thought I already answered that exact question:

> If you mean “exactly one Solr instance in an AZ must host exactly one replica of each shard of the collection”, then yes, that makes sense :).
>
> Okay, one more try :) - here are the rules that should do the trick for you (i.e., what I wrote in the previous sentence):
>
> -----
> rule=shard:*,replica:1,sysprop.AWSAZ:AZ1
> &rule=shard:*,replica:1,sysprop.AWSAZ:AZ2
> &rule=shard:*,replica:1,sysprop.AWSAZ:AZ3
> -----

Have you tried ^^ ?

--
Steve
www.lucidworks.com

Reply | Threaded
Open this post in threaded view
|

Re: Rule-based replication or sharing

Chuck Reynolds
Steve,

Sorry must have omitted it from a past response.

Here is what came back from the response.


<response>
<lst name="responseHeader"><int name="status">400</int><int name="QTime">91</int></lst><str name="Operation create caused exception:">org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: Could not identify nodes matching the rules [{
  "shard":"*",
  "replica":"1",
  "sysprop.AWSAZ":"AZ1"}, {
  "shard":"*",
  "replica":"1",
  "sysprop.AWSAZ":"AZ2"}, {
  "shard":"*",
  "replica":"1",
  "sysprop.AWSAZ":"AZ3"}]
 tag values{
  "10.157.112.223:10002_solr":{"sysprop.AWSAZ":"AZ1"},
  "10.157.120.207:10003_solr":{"sysprop.AWSAZ":"AZ3"},
  "10.157.121.165:10002_solr":{"sysprop.AWSAZ":"AZ3"},
  "10.157.116.190:10002_solr":{"sysprop.AWSAZ":"AZ2"},
  "10.157.121.165:10003_solr":{"sysprop.AWSAZ":"AZ3"},
  "10.157.116.190:10001_solr":{"sysprop.AWSAZ":"AZ2"},
  "10.157.115.30:10003_solr":{"sysprop.AWSAZ":"AZ1"},
  "10.157.121.165:10001_solr":{"sysprop.AWSAZ":"AZ3"},
  "10.157.116.201:10002_solr":{"sysprop.AWSAZ":"AZ2"},
  "10.157.120.207:10001_solr":{"sysprop.AWSAZ":"AZ3"},
  "10.157.112.223:10003_solr":{"sysprop.AWSAZ":"AZ1"},
  "10.157.115.30:10001_solr":{"sysprop.AWSAZ":"AZ1"},
  "10.157.116.190:10003_solr":{"sysprop.AWSAZ":"AZ2"},
  "10.157.112.223:10001_solr":{"sysprop.AWSAZ":"AZ1"},
  "10.157.120.207:10002_solr":{"sysprop.AWSAZ":"AZ3"},
  "10.157.116.201:10003_solr":{"sysprop.AWSAZ":"AZ2"},
  "10.157.116.201:10001_solr":{"sysprop.AWSAZ":"AZ2"},
  "10.157.115.30:10002_solr":{"sysprop.AWSAZ":"AZ1"}}</str><lst name="exception"><str name="msg">Could not identify nodes matching the rules [{
  "shard":"*",
  "replica":"1",
  "sysprop.AWSAZ":"AZ1"}, {
  "shard":"*",
  "replica":"1",
  "sysprop.AWSAZ":"AZ2"}, {
  "shard":"*",
  "replica":"1",
  "sysprop.AWSAZ":"AZ3"}]
 tag values{
  "10.157.112.223:10002_solr":{"sysprop.AWSAZ":"AZ1"},
  "10.157.120.207:10003_solr":{"sysprop.AWSAZ":"AZ3"},
  "10.157.121.165:10002_solr":{"sysprop.AWSAZ":"AZ3"},
  "10.157.116.190:10002_solr":{"sysprop.AWSAZ":"AZ2"},
  "10.157.121.165:10003_solr":{"sysprop.AWSAZ":"AZ3"},
  "10.157.116.190:10001_solr":{"sysprop.AWSAZ":"AZ2"},
  "10.157.115.30:10003_solr":{"sysprop.AWSAZ":"AZ1"},
  "10.157.121.165:10001_solr":{"sysprop.AWSAZ":"AZ3"},
  "10.157.116.201:10002_solr":{"sysprop.AWSAZ":"AZ2"},
  "10.157.120.207:10001_solr":{"sysprop.AWSAZ":"AZ3"},
  "10.157.112.223:10003_solr":{"sysprop.AWSAZ":"AZ1"},
  "10.157.115.30:10001_solr":{"sysprop.AWSAZ":"AZ1"},
  "10.157.116.190:10003_solr":{"sysprop.AWSAZ":"AZ2"},
  "10.157.112.223:10001_solr":{"sysprop.AWSAZ":"AZ1"},
  "10.157.120.207:10002_solr":{"sysprop.AWSAZ":"AZ3"},
  "10.157.116.201:10003_solr":{"sysprop.AWSAZ":"AZ2"},
  "10.157.116.201:10001_solr":{"sysprop.AWSAZ":"AZ2"},
  "10.157.115.30:10002_solr":{"sysprop.AWSAZ":"AZ1"}}</str><int name="rspCode">400</int></lst><lst name="error"><lst name="metadata"><str name="error-class">org.apache.solr.common.SolrException</str><str name="root-error-class">org.apache.solr.common.SolrException</str></lst><str name="msg">Could not identify nodes matching the rules [{
  "shard":"*",
  "replica":"1",
  "sysprop.AWSAZ":"AZ1"}, {
  "shard":"*",
  "replica":"1",
  "sysprop.AWSAZ":"AZ2"}, {
  "shard":"*",
  "replica":"1",
  "sysprop.AWSAZ":"AZ3"}]
 tag values{
  "10.157.112.223:10002_solr":{"sysprop.AWSAZ":"AZ1"},
  "10.157.120.207:10003_solr":{"sysprop.AWSAZ":"AZ3"},
  "10.157.121.165:10002_solr":{"sysprop.AWSAZ":"AZ3"},
  "10.157.116.190:10002_solr":{"sysprop.AWSAZ":"AZ2"},
  "10.157.121.165:10003_solr":{"sysprop.AWSAZ":"AZ3"},
  "10.157.116.190:10001_solr":{"sysprop.AWSAZ":"AZ2"},
  "10.157.115.30:10003_solr":{"sysprop.AWSAZ":"AZ1"},
  "10.157.121.165:10001_solr":{"sysprop.AWSAZ":"AZ3"},
  "10.157.116.201:10002_solr":{"sysprop.AWSAZ":"AZ2"},
  "10.157.120.207:10001_solr":{"sysprop.AWSAZ":"AZ3"},
  "10.157.112.223:10003_solr":{"sysprop.AWSAZ":"AZ1"},
  "10.157.115.30:10001_solr":{"sysprop.AWSAZ":"AZ1"},
  "10.157.116.190:10003_solr":{"sysprop.AWSAZ":"AZ2"},
  "10.157.112.223:10001_solr":{"sysprop.AWSAZ":"AZ1"},
  "10.157.120.207:10002_solr":{"sysprop.AWSAZ":"AZ3"},
  "10.157.116.201:10003_solr":{"sysprop.AWSAZ":"AZ2"},
  "10.157.116.201:10001_solr":{"sysprop.AWSAZ":"AZ2"},
  "10.157.115.30:10002_solr":{"sysprop.AWSAZ":"AZ1"}}</str><int name="code">400</int></lst>
</response>





On 9/25/18, 11:33 AM, "Steve Rowe" <[hidden email]> wrote:

    Chuck, see my responses inline below:
   
    > On Sep 25, 2018, at 12:50 PM, Chuck Reynolds <[hidden email]> wrote:
    > The bottom line is I guess I'm confused by the documentation and the reference to replicas. Normally when referring to replicas in the documentation it is referring to the number of times you want the data replicated. As in replication factor.  That's where the confusion was for me.
   
    We can always use help improving Solr’s documentation, and your perspective is valuable.  Please see https://urldefense.proofpoint.com/v2/url?u=https-3A__wiki.apache.org_solr_HowToContribute&d=DwIFaQ&c=kKqjBR9KKWaWpMhASkPbOg&r=J-2s3b-3-OTA0o6bGDhJXAQlB5Y3s4rOUxlh_78DJl0&m=_ZWThlXl48Sa2f_pVyPzwxiCmVnOtDdddq8wfK6CVqM&s=XR2tqAPyyNaSPvrlnjhqm81DZj3WP3v7s8iEciH1xss&e= and open JIRA issues with the problems you find, and ideally with patches against the ref guide sources.
   
    From the Solr 6.4 ref guide’s “Solr Glossary”:
   
      Replica: A Core that acts as a physical copy of a Shard in a SolrCloud Collection.
   
    As ^^ indicates, a “replica” is not a replication factor.  (Though “replica:1” in a rule-based replica placement rule is a condition on replica *count*, so I can see where that could be confusing.)
   
    > If I want to create a rule that insures that my replication factor of three correctly shards the data across three AZ so if I was to lose one or even two AZ's in AWS Solr would still have 1 - 2 copies of the data.   How would that rule work?
   
    I thought I already answered that exact question:
   
    > If you mean “exactly one Solr instance in an AZ must host exactly one replica of each shard of the collection”, then yes, that makes sense :).
    >
    > Okay, one more try :) - here are the rules that should do the trick for you (i.e., what I wrote in the previous sentence):
    >
    > -----
    > rule=shard:*,replica:1,sysprop.AWSAZ:AZ1
    > &rule=shard:*,replica:1,sysprop.AWSAZ:AZ2
    > &rule=shard:*,replica:1,sysprop.AWSAZ:AZ3
    > -----
   
    Have you tried ^^ ?
   
    --
    Steve
    https://urldefense.proofpoint.com/v2/url?u=http-3A__www.lucidworks.com&d=DwIFaQ&c=kKqjBR9KKWaWpMhASkPbOg&r=J-2s3b-3-OTA0o6bGDhJXAQlB5Y3s4rOUxlh_78DJl0&m=_ZWThlXl48Sa2f_pVyPzwxiCmVnOtDdddq8wfK6CVqM&s=0npMKzqmYyK1N3r08vl6qYeDyiOMVsr_iXcvW5nNUqk&e=
   
   

Reply | Threaded
Open this post in threaded view
|

Re: Rule-based replication or sharing

Shawn Heisey-2
In reply to this post by Chuck Reynolds
On 9/25/2018 9:21 AM, Chuck Reynolds wrote:
> Each server has three instances of Solr running on it so every instance on the server has to be in the same replica set.

You should be running exactly one Solr instance per server.  When
evaluating rules for replica placement, SolrCloud will treat each
instance as completely separate from all others, including others on the
same machine.  It will not know that those three instances are on the
same machine.  One Solr instance can handle MANY indexes.

There is only ONE situation where it makes sense to run multiple
instances per machine, and in my strong opinion, even that situation
should not be handled with multiple instances. That situation is this: 
When running one instance would require a REALLY large heap.  Garbage
collection pauses can become extreme in that situation, so some people
will run multiple instances that each have a smaller heap, and divide
their indexes between them. In my opinion, when you have enough index
data on an instance that it requires a huge heap, instead of running two
or more instances on one server, it's time to add more servers.

Thanks,
Shawn

Reply | Threaded
Open this post in threaded view
|

Re: Rule-based replication or sharing

Chuck Reynolds
Shawn,

Thanks for the info. We’ve been running this way for the past 4 years.

We were running on very large hardware, 20 physical cores with 256 gigs of ram with 3 billion document and it was the only way we could take advantage of the hardware.

Running 1 Solr instance per server never gave us the throughput we needed.

So I somewhat disagree with your statement because our test proved otherwise.

Thanks for the info.

Sent from my iPhone

> On Sep 25, 2018, at 4:19 PM, Shawn Heisey <[hidden email]> wrote:
>
>> On 9/25/2018 9:21 AM, Chuck Reynolds wrote:
>> Each server has three instances of Solr running on it so every instance on the server has to be in the same replica set.
>
> You should be running exactly one Solr instance per server.  When evaluating rules for replica placement, SolrCloud will treat each instance as completely separate from all others, including others on the same machine.  It will not know that those three instances are on the same machine.  One Solr instance can handle MANY indexes.
>
> There is only ONE situation where it makes sense to run multiple instances per machine, and in my strong opinion, even that situation should not be handled with multiple instances. That situation is this:  When running one instance would require a REALLY large heap.  Garbage collection pauses can become extreme in that situation, so some people will run multiple instances that each have a smaller heap, and divide their indexes between them. In my opinion, when you have enough index data on an instance that it requires a huge heap, instead of running two or more instances on one server, it's time to add more servers.
>
> Thanks,
> Shawn
>
Reply | Threaded
Open this post in threaded view
|

Re: Rule-based replication or sharing

Noble Paul നോബിള്‍  नोब्ळ्
The rules suggested by Steve is correct. I tested it locally and I got
the same errors. That means a bug exists probably.
All the new development efforts are invested in the new policy feature
.https://lucene.apache.org/solr/guide/7_4/solrcloud-autoscaling-policy-preferences.html

The old one is going to be deprecated pretty soon. So, I'm not sure if
we should be investing our resources here
On Wed, Sep 26, 2018 at 1:23 PM Chuck Reynolds <[hidden email]> wrote:

>
> Shawn,
>
> Thanks for the info. We’ve been running this way for the past 4 years.
>
> We were running on very large hardware, 20 physical cores with 256 gigs of ram with 3 billion document and it was the only way we could take advantage of the hardware.
>
> Running 1 Solr instance per server never gave us the throughput we needed.
>
> So I somewhat disagree with your statement because our test proved otherwise.
>
> Thanks for the info.
>
> Sent from my iPhone
>
> > On Sep 25, 2018, at 4:19 PM, Shawn Heisey <[hidden email]> wrote:
> >
> >> On 9/25/2018 9:21 AM, Chuck Reynolds wrote:
> >> Each server has three instances of Solr running on it so every instance on the server has to be in the same replica set.
> >
> > You should be running exactly one Solr instance per server.  When evaluating rules for replica placement, SolrCloud will treat each instance as completely separate from all others, including others on the same machine.  It will not know that those three instances are on the same machine.  One Solr instance can handle MANY indexes.
> >
> > There is only ONE situation where it makes sense to run multiple instances per machine, and in my strong opinion, even that situation should not be handled with multiple instances. That situation is this:  When running one instance would require a REALLY large heap.  Garbage collection pauses can become extreme in that situation, so some people will run multiple instances that each have a smaller heap, and divide their indexes between them. In my opinion, when you have enough index data on an instance that it requires a huge heap, instead of running two or more instances on one server, it's time to add more servers.
> >
> > Thanks,
> > Shawn
> >



--
-----------------------------------------------------
Noble Paul
Reply | Threaded
Open this post in threaded view
|

Re: Rule-based replication or sharing

Chuck Reynolds
Noble,

So other than manually moving replicas of shard do you have a suggestion of how one might accomplish the multiple availability zone with multiple instances of Solr running on each server?

Thanks

On 9/26/18, 12:56 AM, "Noble Paul" <[hidden email]> wrote:

    The rules suggested by Steve is correct. I tested it locally and I got
    the same errors. That means a bug exists probably.
    All the new development efforts are invested in the new policy feature
    .https://urldefense.proofpoint.com/v2/url?u=https-3A__lucene.apache.org_solr_guide_7-5F4_solrcloud-2Dautoscaling-2Dpolicy-2Dpreferences.html&d=DwIFaQ&c=kKqjBR9KKWaWpMhASkPbOg&r=J-2s3b-3-OTA0o6bGDhJXAQlB5Y3s4rOUxlh_78DJl0&m=yXVYNcm-dqN_lucLyuQI38EZfK4f8l4828Ty53e4plM&s=D1vfu3bOu_hOGAU2CIKPwqBTPkYiBeK1kOUoFnQZpKA&e=
   
    The old one is going to be deprecated pretty soon. So, I'm not sure if
    we should be investing our resources here
    On Wed, Sep 26, 2018 at 1:23 PM Chuck Reynolds <[hidden email]> wrote:
    >
    > Shawn,
    >
    > Thanks for the info. We’ve been running this way for the past 4 years.
    >
    > We were running on very large hardware, 20 physical cores with 256 gigs of ram with 3 billion document and it was the only way we could take advantage of the hardware.
    >
    > Running 1 Solr instance per server never gave us the throughput we needed.
    >
    > So I somewhat disagree with your statement because our test proved otherwise.
    >
    > Thanks for the info.
    >
    > Sent from my iPhone
    >
    > > On Sep 25, 2018, at 4:19 PM, Shawn Heisey <[hidden email]> wrote:
    > >
    > >> On 9/25/2018 9:21 AM, Chuck Reynolds wrote:
    > >> Each server has three instances of Solr running on it so every instance on the server has to be in the same replica set.
    > >
    > > You should be running exactly one Solr instance per server.  When evaluating rules for replica placement, SolrCloud will treat each instance as completely separate from all others, including others on the same machine.  It will not know that those three instances are on the same machine.  One Solr instance can handle MANY indexes.
    > >
    > > There is only ONE situation where it makes sense to run multiple instances per machine, and in my strong opinion, even that situation should not be handled with multiple instances. That situation is this:  When running one instance would require a REALLY large heap.  Garbage collection pauses can become extreme in that situation, so some people will run multiple instances that each have a smaller heap, and divide their indexes between them. In my opinion, when you have enough index data on an instance that it requires a huge heap, instead of running two or more instances on one server, it's time to add more servers.
    > >
    > > Thanks,
    > > Shawn
    > >
   
   
   
    --
    -----------------------------------------------------
    Noble Paul
   

Reply | Threaded
Open this post in threaded view
|

Re: Rule-based replication or sharing

Noble Paul നോബിള്‍  नोब्ळ्
I'm not sure if it is pertinent to ask you to move to the latest Solr
which has the policy based replica placement. Unfortunately, I don't
have any other solution I can think of

On Wed, Sep 26, 2018 at 11:46 PM Chuck Reynolds <[hidden email]> wrote:

>
> Noble,
>
> So other than manually moving replicas of shard do you have a suggestion of how one might accomplish the multiple availability zone with multiple instances of Solr running on each server?
>
> Thanks
>
> On 9/26/18, 12:56 AM, "Noble Paul" <[hidden email]> wrote:
>
>     The rules suggested by Steve is correct. I tested it locally and I got
>     the same errors. That means a bug exists probably.
>     All the new development efforts are invested in the new policy feature
>     .https://urldefense.proofpoint.com/v2/url?u=https-3A__lucene.apache.org_solr_guide_7-5F4_solrcloud-2Dautoscaling-2Dpolicy-2Dpreferences.html&d=DwIFaQ&c=kKqjBR9KKWaWpMhASkPbOg&r=J-2s3b-3-OTA0o6bGDhJXAQlB5Y3s4rOUxlh_78DJl0&m=yXVYNcm-dqN_lucLyuQI38EZfK4f8l4828Ty53e4plM&s=D1vfu3bOu_hOGAU2CIKPwqBTPkYiBeK1kOUoFnQZpKA&e=
>
>     The old one is going to be deprecated pretty soon. So, I'm not sure if
>     we should be investing our resources here
>     On Wed, Sep 26, 2018 at 1:23 PM Chuck Reynolds <[hidden email]> wrote:
>     >
>     > Shawn,
>     >
>     > Thanks for the info. We’ve been running this way for the past 4 years.
>     >
>     > We were running on very large hardware, 20 physical cores with 256 gigs of ram with 3 billion document and it was the only way we could take advantage of the hardware.
>     >
>     > Running 1 Solr instance per server never gave us the throughput we needed.
>     >
>     > So I somewhat disagree with your statement because our test proved otherwise.
>     >
>     > Thanks for the info.
>     >
>     > Sent from my iPhone
>     >
>     > > On Sep 25, 2018, at 4:19 PM, Shawn Heisey <[hidden email]> wrote:
>     > >
>     > >> On 9/25/2018 9:21 AM, Chuck Reynolds wrote:
>     > >> Each server has three instances of Solr running on it so every instance on the server has to be in the same replica set.
>     > >
>     > > You should be running exactly one Solr instance per server.  When evaluating rules for replica placement, SolrCloud will treat each instance as completely separate from all others, including others on the same machine.  It will not know that those three instances are on the same machine.  One Solr instance can handle MANY indexes.
>     > >
>     > > There is only ONE situation where it makes sense to run multiple instances per machine, and in my strong opinion, even that situation should not be handled with multiple instances. That situation is this:  When running one instance would require a REALLY large heap.  Garbage collection pauses can become extreme in that situation, so some people will run multiple instances that each have a smaller heap, and divide their indexes between them. In my opinion, when you have enough index data on an instance that it requires a huge heap, instead of running two or more instances on one server, it's time to add more servers.
>     > >
>     > > Thanks,
>     > > Shawn
>     > >
>
>
>
>     --
>     -----------------------------------------------------
>     Noble Paul
>
>


--
-----------------------------------------------------
Noble Paul
Reply | Threaded
Open this post in threaded view
|

Re: Rule-based replication or sharing

Chuck Reynolds
Noble,

Are you saying in the latest version of Solr that this would work with three instances of Solr running on each server?

If so how?

Thanks again for your help.

On 9/26/18, 9:11 AM, "Noble Paul" <[hidden email]> wrote:

    I'm not sure if it is pertinent to ask you to move to the latest Solr
    which has the policy based replica placement. Unfortunately, I don't
    have any other solution I can think of
   
    On Wed, Sep 26, 2018 at 11:46 PM Chuck Reynolds <[hidden email]> wrote:
    >
    > Noble,
    >
    > So other than manually moving replicas of shard do you have a suggestion of how one might accomplish the multiple availability zone with multiple instances of Solr running on each server?
    >
    > Thanks
    >
    > On 9/26/18, 12:56 AM, "Noble Paul" <[hidden email]> wrote:
    >
    >     The rules suggested by Steve is correct. I tested it locally and I got
    >     the same errors. That means a bug exists probably.
    >     All the new development efforts are invested in the new policy feature
    >     .https://urldefense.proofpoint.com/v2/url?u=https-3A__lucene.apache.org_solr_guide_7-5F4_solrcloud-2Dautoscaling-2Dpolicy-2Dpreferences.html&d=DwIFaQ&c=kKqjBR9KKWaWpMhASkPbOg&r=J-2s3b-3-OTA0o6bGDhJXAQlB5Y3s4rOUxlh_78DJl0&m=yXVYNcm-dqN_lucLyuQI38EZfK4f8l4828Ty53e4plM&s=D1vfu3bOu_hOGAU2CIKPwqBTPkYiBeK1kOUoFnQZpKA&e=
    >
    >     The old one is going to be deprecated pretty soon. So, I'm not sure if
    >     we should be investing our resources here
    >     On Wed, Sep 26, 2018 at 1:23 PM Chuck Reynolds <[hidden email]> wrote:
    >     >
    >     > Shawn,
    >     >
    >     > Thanks for the info. We’ve been running this way for the past 4 years.
    >     >
    >     > We were running on very large hardware, 20 physical cores with 256 gigs of ram with 3 billion document and it was the only way we could take advantage of the hardware.
    >     >
    >     > Running 1 Solr instance per server never gave us the throughput we needed.
    >     >
    >     > So I somewhat disagree with your statement because our test proved otherwise.
    >     >
    >     > Thanks for the info.
    >     >
    >     > Sent from my iPhone
    >     >
    >     > > On Sep 25, 2018, at 4:19 PM, Shawn Heisey <[hidden email]> wrote:
    >     > >
    >     > >> On 9/25/2018 9:21 AM, Chuck Reynolds wrote:
    >     > >> Each server has three instances of Solr running on it so every instance on the server has to be in the same replica set.
    >     > >
    >     > > You should be running exactly one Solr instance per server.  When evaluating rules for replica placement, SolrCloud will treat each instance as completely separate from all others, including others on the same machine.  It will not know that those three instances are on the same machine.  One Solr instance can handle MANY indexes.
    >     > >
    >     > > There is only ONE situation where it makes sense to run multiple instances per machine, and in my strong opinion, even that situation should not be handled with multiple instances. That situation is this:  When running one instance would require a REALLY large heap.  Garbage collection pauses can become extreme in that situation, so some people will run multiple instances that each have a smaller heap, and divide their indexes between them. In my opinion, when you have enough index data on an instance that it requires a huge heap, instead of running two or more instances on one server, it's time to add more servers.
    >     > >
    >     > > Thanks,
    >     > > Shawn
    >     > >
    >
    >
    >
    >     --
    >     -----------------------------------------------------
    >     Noble Paul
    >
    >
   
   
    --
    -----------------------------------------------------
    Noble Paul
   

Reply | Threaded
Open this post in threaded view
|

Re: Rule-based replication or sharing

Noble Paul നോബിള്‍  नोब्ळ्
Yes, it uses a the autoscaling policies to achieve the same. Please refer
to the documentation here
https://lucene.apache.org/solr/guide/7_5/solrcloud-autoscaling-policy-preferences.html

On Thu, Sep 27, 2018, 02:11 Chuck Reynolds <[hidden email]> wrote:

> Noble,
>
> Are you saying in the latest version of Solr that this would work with
> three instances of Solr running on each server?
>
> If so how?
>
> Thanks again for your help.
>
> On 9/26/18, 9:11 AM, "Noble Paul" <[hidden email]> wrote:
>
>     I'm not sure if it is pertinent to ask you to move to the latest Solr
>     which has the policy based replica placement. Unfortunately, I don't
>     have any other solution I can think of
>
>     On Wed, Sep 26, 2018 at 11:46 PM Chuck Reynolds <
> [hidden email]> wrote:
>     >
>     > Noble,
>     >
>     > So other than manually moving replicas of shard do you have a
> suggestion of how one might accomplish the multiple availability zone with
> multiple instances of Solr running on each server?
>     >
>     > Thanks
>     >
>     > On 9/26/18, 12:56 AM, "Noble Paul" <[hidden email]> wrote:
>     >
>     >     The rules suggested by Steve is correct. I tested it locally and
> I got
>     >     the same errors. That means a bug exists probably.
>     >     All the new development efforts are invested in the new policy
> feature
>     >     .
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lucene.apache.org_solr_guide_7-5F4_solrcloud-2Dautoscaling-2Dpolicy-2Dpreferences.html&d=DwIFaQ&c=kKqjBR9KKWaWpMhASkPbOg&r=J-2s3b-3-OTA0o6bGDhJXAQlB5Y3s4rOUxlh_78DJl0&m=yXVYNcm-dqN_lucLyuQI38EZfK4f8l4828Ty53e4plM&s=D1vfu3bOu_hOGAU2CIKPwqBTPkYiBeK1kOUoFnQZpKA&e=
>     >
>     >     The old one is going to be deprecated pretty soon. So, I'm not
> sure if
>     >     we should be investing our resources here
>     >     On Wed, Sep 26, 2018 at 1:23 PM Chuck Reynolds <
> [hidden email]> wrote:
>     >     >
>     >     > Shawn,
>     >     >
>     >     > Thanks for the info. We’ve been running this way for the past
> 4 years.
>     >     >
>     >     > We were running on very large hardware, 20 physical cores with
> 256 gigs of ram with 3 billion document and it was the only way we could
> take advantage of the hardware.
>     >     >
>     >     > Running 1 Solr instance per server never gave us the
> throughput we needed.
>     >     >
>     >     > So I somewhat disagree with your statement because our test
> proved otherwise.
>     >     >
>     >     > Thanks for the info.
>     >     >
>     >     > Sent from my iPhone
>     >     >
>     >     > > On Sep 25, 2018, at 4:19 PM, Shawn Heisey <
> [hidden email]> wrote:
>     >     > >
>     >     > >> On 9/25/2018 9:21 AM, Chuck Reynolds wrote:
>     >     > >> Each server has three instances of Solr running on it so
> every instance on the server has to be in the same replica set.
>     >     > >
>     >     > > You should be running exactly one Solr instance per server.
> When evaluating rules for replica placement, SolrCloud will treat each
> instance as completely separate from all others, including others on the
> same machine.  It will not know that those three instances are on the same
> machine.  One Solr instance can handle MANY indexes.
>     >     > >
>     >     > > There is only ONE situation where it makes sense to run
> multiple instances per machine, and in my strong opinion, even that
> situation should not be handled with multiple instances. That situation is
> this:  When running one instance would require a REALLY large heap.
> Garbage collection pauses can become extreme in that situation, so some
> people will run multiple instances that each have a smaller heap, and
> divide their indexes between them. In my opinion, when you have enough
> index data on an instance that it requires a huge heap, instead of running
> two or more instances on one server, it's time to add more servers.
>     >     > >
>     >     > > Thanks,
>     >     > > Shawn
>     >     > >
>     >
>     >
>     >
>     >     --
>     >     -----------------------------------------------------
>     >     Noble Paul
>     >
>     >
>
>
>     --
>     -----------------------------------------------------
>     Noble Paul
>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Rule-based replication or sharing

Varun Thacker-4
In reply to this post by Chuck Reynolds
Hi Chuck,

I was chatting with Noble offline and he suggested we could use this
starting 7.5

*{replica:'#EQUAL', shard : ''#EACH' , sysprop.az <http://sysprop.az>
:'#EACH'}*

where "az" is a sysprop while starting each solr instance ( -Daz=us-east-1 )

It's documented
https://lucene.apache.org/solr/guide/7_5/solrcloud-autoscaling-policy-preferences.html

Let me know if this works for you.


On Wed, Sep 26, 2018 at 9:11 AM Chuck Reynolds <[hidden email]>
wrote:

> Noble,
>
> Are you saying in the latest version of Solr that this would work with
> three instances of Solr running on each server?
>
> If so how?
>
> Thanks again for your help.
>
> On 9/26/18, 9:11 AM, "Noble Paul" <[hidden email]> wrote:
>
>     I'm not sure if it is pertinent to ask you to move to the latest Solr
>     which has the policy based replica placement. Unfortunately, I don't
>     have any other solution I can think of
>
>     On Wed, Sep 26, 2018 at 11:46 PM Chuck Reynolds <
> [hidden email]> wrote:
>     >
>     > Noble,
>     >
>     > So other than manually moving replicas of shard do you have a
> suggestion of how one might accomplish the multiple availability zone with
> multiple instances of Solr running on each server?
>     >
>     > Thanks
>     >
>     > On 9/26/18, 12:56 AM, "Noble Paul" <[hidden email]> wrote:
>     >
>     >     The rules suggested by Steve is correct. I tested it locally and
> I got
>     >     the same errors. That means a bug exists probably.
>     >     All the new development efforts are invested in the new policy
> feature
>     >     .
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lucene.apache.org_solr_guide_7-5F4_solrcloud-2Dautoscaling-2Dpolicy-2Dpreferences.html&d=DwIFaQ&c=kKqjBR9KKWaWpMhASkPbOg&r=J-2s3b-3-OTA0o6bGDhJXAQlB5Y3s4rOUxlh_78DJl0&m=yXVYNcm-dqN_lucLyuQI38EZfK4f8l4828Ty53e4plM&s=D1vfu3bOu_hOGAU2CIKPwqBTPkYiBeK1kOUoFnQZpKA&e=
>     >
>     >     The old one is going to be deprecated pretty soon. So, I'm not
> sure if
>     >     we should be investing our resources here
>     >     On Wed, Sep 26, 2018 at 1:23 PM Chuck Reynolds <
> [hidden email]> wrote:
>     >     >
>     >     > Shawn,
>     >     >
>     >     > Thanks for the info. We’ve been running this way for the past
> 4 years.
>     >     >
>     >     > We were running on very large hardware, 20 physical cores with
> 256 gigs of ram with 3 billion document and it was the only way we could
> take advantage of the hardware.
>     >     >
>     >     > Running 1 Solr instance per server never gave us the
> throughput we needed.
>     >     >
>     >     > So I somewhat disagree with your statement because our test
> proved otherwise.
>     >     >
>     >     > Thanks for the info.
>     >     >
>     >     > Sent from my iPhone
>     >     >
>     >     > > On Sep 25, 2018, at 4:19 PM, Shawn Heisey <
> [hidden email]> wrote:
>     >     > >
>     >     > >> On 9/25/2018 9:21 AM, Chuck Reynolds wrote:
>     >     > >> Each server has three instances of Solr running on it so
> every instance on the server has to be in the same replica set.
>     >     > >
>     >     > > You should be running exactly one Solr instance per server.
> When evaluating rules for replica placement, SolrCloud will treat each
> instance as completely separate from all others, including others on the
> same machine.  It will not know that those three instances are on the same
> machine.  One Solr instance can handle MANY indexes.
>     >     > >
>     >     > > There is only ONE situation where it makes sense to run
> multiple instances per machine, and in my strong opinion, even that
> situation should not be handled with multiple instances. That situation is
> this:  When running one instance would require a REALLY large heap.
> Garbage collection pauses can become extreme in that situation, so some
> people will run multiple instances that each have a smaller heap, and
> divide their indexes between them. In my opinion, when you have enough
> index data on an instance that it requires a huge heap, instead of running
> two or more instances on one server, it's time to add more servers.
>     >     > >
>     >     > > Thanks,
>     >     > > Shawn
>     >     > >
>     >
>     >
>     >
>     >     --
>     >     -----------------------------------------------------
>     >     Noble Paul
>     >
>     >
>
>
>     --
>     -----------------------------------------------------
>     Noble Paul
>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Rule-based replication or sharing

Varun Thacker-4
Hi Chuck,

I was chatting with Noble offline and he suggested we could use this
starting 7.5

{replica:'#EQUAL', shard : ''#EACH' , sysprop.az :'#EACH'}

where "az" is a sysprop while starting each solr instance ( -Daz=us-east-1 )

It's documented
https://lucene.apache.org/solr/guide/7_5/solrcloud-autoscaling-policy-preferences.html

Let me know if this works for you.

( Looks like my previous email had some formatting issues )

On Mon, Oct 1, 2018 at 10:17 PM Varun Thacker <[hidden email]> wrote:

> Hi Chuck,
>
> I was chatting with Noble offline and he suggested we could use this
> starting 7.5
>
> *{replica:'#EQUAL', shard : ''#EACH' , sysprop.az <http://sysprop.az>
> :'#EACH'}*
>
> where "az" is a sysprop while starting each solr instance ( -Daz=us-east-1
> )
>
> It's documented
> https://lucene.apache.org/solr/guide/7_5/solrcloud-autoscaling-policy-preferences.html
>
> Let me know if this works for you.
>
>
> On Wed, Sep 26, 2018 at 9:11 AM Chuck Reynolds <[hidden email]>
> wrote:
>
>> Noble,
>>
>> Are you saying in the latest version of Solr that this would work with
>> three instances of Solr running on each server?
>>
>> If so how?
>>
>> Thanks again for your help.
>>
>> On 9/26/18, 9:11 AM, "Noble Paul" <[hidden email]> wrote:
>>
>>     I'm not sure if it is pertinent to ask you to move to the latest Solr
>>     which has the policy based replica placement. Unfortunately, I don't
>>     have any other solution I can think of
>>
>>     On Wed, Sep 26, 2018 at 11:46 PM Chuck Reynolds <
>> [hidden email]> wrote:
>>     >
>>     > Noble,
>>     >
>>     > So other than manually moving replicas of shard do you have a
>> suggestion of how one might accomplish the multiple availability zone with
>> multiple instances of Solr running on each server?
>>     >
>>     > Thanks
>>     >
>>     > On 9/26/18, 12:56 AM, "Noble Paul" <[hidden email]> wrote:
>>     >
>>     >     The rules suggested by Steve is correct. I tested it locally
>> and I got
>>     >     the same errors. That means a bug exists probably.
>>     >     All the new development efforts are invested in the new policy
>> feature
>>     >     .
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lucene.apache.org_solr_guide_7-5F4_solrcloud-2Dautoscaling-2Dpolicy-2Dpreferences.html&d=DwIFaQ&c=kKqjBR9KKWaWpMhASkPbOg&r=J-2s3b-3-OTA0o6bGDhJXAQlB5Y3s4rOUxlh_78DJl0&m=yXVYNcm-dqN_lucLyuQI38EZfK4f8l4828Ty53e4plM&s=D1vfu3bOu_hOGAU2CIKPwqBTPkYiBeK1kOUoFnQZpKA&e=
>>     >
>>     >     The old one is going to be deprecated pretty soon. So, I'm not
>> sure if
>>     >     we should be investing our resources here
>>     >     On Wed, Sep 26, 2018 at 1:23 PM Chuck Reynolds <
>> [hidden email]> wrote:
>>     >     >
>>     >     > Shawn,
>>     >     >
>>     >     > Thanks for the info. We’ve been running this way for the past
>> 4 years.
>>     >     >
>>     >     > We were running on very large hardware, 20 physical cores
>> with 256 gigs of ram with 3 billion document and it was the only way we
>> could take advantage of the hardware.
>>     >     >
>>     >     > Running 1 Solr instance per server never gave us the
>> throughput we needed.
>>     >     >
>>     >     > So I somewhat disagree with your statement because our test
>> proved otherwise.
>>     >     >
>>     >     > Thanks for the info.
>>     >     >
>>     >     > Sent from my iPhone
>>     >     >
>>     >     > > On Sep 25, 2018, at 4:19 PM, Shawn Heisey <
>> [hidden email]> wrote:
>>     >     > >
>>     >     > >> On 9/25/2018 9:21 AM, Chuck Reynolds wrote:
>>     >     > >> Each server has three instances of Solr running on it so
>> every instance on the server has to be in the same replica set.
>>     >     > >
>>     >     > > You should be running exactly one Solr instance per
>> server.  When evaluating rules for replica placement, SolrCloud will treat
>> each instance as completely separate from all others, including others on
>> the same machine.  It will not know that those three instances are on the
>> same machine.  One Solr instance can handle MANY indexes.
>>     >     > >
>>     >     > > There is only ONE situation where it makes sense to run
>> multiple instances per machine, and in my strong opinion, even that
>> situation should not be handled with multiple instances. That situation is
>> this:  When running one instance would require a REALLY large heap.
>> Garbage collection pauses can become extreme in that situation, so some
>> people will run multiple instances that each have a smaller heap, and
>> divide their indexes between them. In my opinion, when you have enough
>> index data on an instance that it requires a huge heap, instead of running
>> two or more instances on one server, it's time to add more servers.
>>     >     > >
>>     >     > > Thanks,
>>     >     > > Shawn
>>     >     > >
>>     >
>>     >
>>     >
>>     >     --
>>     >     -----------------------------------------------------
>>     >     Noble Paul
>>     >
>>     >
>>
>>
>>     --
>>     -----------------------------------------------------
>>     Noble Paul
>>
>>
>>
12