How do we interpret replicationFactor ?

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

How do we interpret replicationFactor ?

Varun Thacker-4
While working on SOLR-11676 a few questions came that were't obvious

Should a user be allowed to specify replicationFactor and nrtReplicas ? Today it's possible but my answer was it shouldn't be. What do others think? 

If everyone agrees the two shouldn't be specified then there is one problem while fixing this - SolrJ

if (nrtReplicas != null) {
params.set( ZkStateReader.REPLICATION_FACTOR, nrtReplicas);// Keep both for compatibility?
params.set( ZkStateReader.NRT_REPLICAS, nrtReplicas);
}
SolrJ sets both replicationFactor and nrtReplicas with the same value. So if we simply put a check at the server saying "don't allow both parameters" all SolrJ calls from older clients will fail 
The compromise would be the server could check if both nrtReplicas and replicationFactor are equal then don't error out

Second question, SolrJ doesn't allow a user to specify replicationFactor but if you're using the API directly it's allowed.
Do we plan on deprecating replicationFactor eventually in favour of nrtReplicas ? If yes would 7.5 be a good place to start throwing a warning ? 

Reply | Threaded
Open this post in threaded view
|

Re: How do we interpret replicationFactor ?

Tomás Fernández Löbbe
I think we should deprecate it. There were some concerns about this because new users would understand quickly what "replicationFactor" is, while "nrtReplicas" is not so intuitive, but I don't like having two ways to do the same, and now that there are multiple types of replicas I think it's better for the parameter to be explicit.
I would still keep accepting the parameter for backwards compatibility, but maybe remove the internal use of it? Maybe even remove it from the clusterstate (and again, make sure we can still read cluster states that have it, for upgrades).

On Thu, Jun 14, 2018 at 2:46 PM, Varun Thacker <[hidden email]> wrote:
While working on SOLR-11676 a few questions came that were't obvious

Should a user be allowed to specify replicationFactor and nrtReplicas ? Today it's possible but my answer was it shouldn't be. What do others think? 

If everyone agrees the two shouldn't be specified then there is one problem while fixing this - SolrJ

if (nrtReplicas != null) {
params.set( ZkStateReader.REPLICATION_FACTOR, nrtReplicas);// Keep both for compatibility?
params.set( ZkStateReader.NRT_REPLICAS, nrtReplicas);
}
SolrJ sets both replicationFactor and nrtReplicas with the same value. So if we simply put a check at the server saying "don't allow both parameters" all SolrJ calls from older clients will fail 
The compromise would be the server could check if both nrtReplicas and replicationFactor are equal then don't error out

Second question, SolrJ doesn't allow a user to specify replicationFactor but if you're using the API directly it's allowed.
Do we plan on deprecating replicationFactor eventually in favour of nrtReplicas ? If yes would 7.5 be a good place to start throwing a warning ? 


Reply | Threaded
Open this post in threaded view
|

Re: How do we interpret replicationFactor ?

Varun Thacker-4
Thanks Tomás

The approach I'm taking is SolrJ never sets replicationFactor and keep back-compat for older clients who would set both replicationFactor and nrtReplicas for the same thing

I'm not going to remove it from cluster state just yet ( even with keeping back-compat ) . I'm thinking this parameter could mean an overarching replicationFactor ( used internally ) which would be a sum of all the replica types . We could use this info internally while external users would not be able to set it in the future 

On Fri, Jun 15, 2018 at 10:06 AM, Tomás Fernández Löbbe <[hidden email]> wrote:
I think we should deprecate it. There were some concerns about this because new users would understand quickly what "replicationFactor" is, while "nrtReplicas" is not so intuitive, but I don't like having two ways to do the same, and now that there are multiple types of replicas I think it's better for the parameter to be explicit.
I would still keep accepting the parameter for backwards compatibility, but maybe remove the internal use of it? Maybe even remove it from the clusterstate (and again, make sure we can still read cluster states that have it, for upgrades).

On Thu, Jun 14, 2018 at 2:46 PM, Varun Thacker <[hidden email]> wrote:
While working on SOLR-11676 a few questions came that were't obvious

Should a user be allowed to specify replicationFactor and nrtReplicas ? Today it's possible but my answer was it shouldn't be. What do others think? 

If everyone agrees the two shouldn't be specified then there is one problem while fixing this - SolrJ

if (nrtReplicas != null) {
params.set( ZkStateReader.REPLICATION_FACTOR, nrtReplicas);// Keep both for compatibility?
params.set( ZkStateReader.NRT_REPLICAS, nrtReplicas);
}
SolrJ sets both replicationFactor and nrtReplicas with the same value. So if we simply put a check at the server saying "don't allow both parameters" all SolrJ calls from older clients will fail 
The compromise would be the server could check if both nrtReplicas and replicationFactor are equal then don't error out

Second question, SolrJ doesn't allow a user to specify replicationFactor but if you're using the API directly it's allowed.
Do we plan on deprecating replicationFactor eventually in favour of nrtReplicas ? If yes would 7.5 be a good place to start throwing a warning ? 



Reply | Threaded
Open this post in threaded view
|

Re: How do we interpret replicationFactor ?

Varun Thacker-4
If someone has 5 minutes could they please review my approach taken in SOLR-11676

On Fri, Jun 15, 2018 at 12:24 PM, Varun Thacker <[hidden email]> wrote:
Thanks Tomás

The approach I'm taking is SolrJ never sets replicationFactor and keep back-compat for older clients who would set both replicationFactor and nrtReplicas for the same thing

I'm not going to remove it from cluster state just yet ( even with keeping back-compat ) . I'm thinking this parameter could mean an overarching replicationFactor ( used internally ) which would be a sum of all the replica types . We could use this info internally while external users would not be able to set it in the future 

On Fri, Jun 15, 2018 at 10:06 AM, Tomás Fernández Löbbe <[hidden email]> wrote:
I think we should deprecate it. There were some concerns about this because new users would understand quickly what "replicationFactor" is, while "nrtReplicas" is not so intuitive, but I don't like having two ways to do the same, and now that there are multiple types of replicas I think it's better for the parameter to be explicit.
I would still keep accepting the parameter for backwards compatibility, but maybe remove the internal use of it? Maybe even remove it from the clusterstate (and again, make sure we can still read cluster states that have it, for upgrades).

On Thu, Jun 14, 2018 at 2:46 PM, Varun Thacker <[hidden email]> wrote:
While working on SOLR-11676 a few questions came that were't obvious

Should a user be allowed to specify replicationFactor and nrtReplicas ? Today it's possible but my answer was it shouldn't be. What do others think? 

If everyone agrees the two shouldn't be specified then there is one problem while fixing this - SolrJ

if (nrtReplicas != null) {
params.set( ZkStateReader.REPLICATION_FACTOR, nrtReplicas);// Keep both for compatibility?
params.set( ZkStateReader.NRT_REPLICAS, nrtReplicas);
}
SolrJ sets both replicationFactor and nrtReplicas with the same value. So if we simply put a check at the server saying "don't allow both parameters" all SolrJ calls from older clients will fail 
The compromise would be the server could check if both nrtReplicas and replicationFactor are equal then don't error out

Second question, SolrJ doesn't allow a user to specify replicationFactor but if you're using the API directly it's allowed.
Do we plan on deprecating replicationFactor eventually in favour of nrtReplicas ? If yes would 7.5 be a good place to start throwing a warning ?