[jira] [Created] (SOLR-2752) leader-per-shard

classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

[jira] [Created] (SOLR-2752) leader-per-shard

Sebastian Nagel (Jira)
leader-per-shard
----------------

                 Key: SOLR-2752
                 URL: https://issues.apache.org/jira/browse/SOLR-2752
             Project: Solr
          Issue Type: Sub-task
          Components: SolrCloud
            Reporter: Yonik Seeley
             Fix For: 4.0


We need to add metadata into zookeeper about who is the leader for each shard, and have some kind of leader election.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] [Assigned] (SOLR-2752) leader-per-shard

Sebastian Nagel (Jira)

     [ https://issues.apache.org/jira/browse/SOLR-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mark Miller reassigned SOLR-2752:
---------------------------------

    Assignee: Mark Miller

> leader-per-shard
> ----------------
>
>                 Key: SOLR-2752
>                 URL: https://issues.apache.org/jira/browse/SOLR-2752
>             Project: Solr
>          Issue Type: Sub-task
>          Components: SolrCloud
>            Reporter: Yonik Seeley
>            Assignee: Mark Miller
>             Fix For: 4.0
>
>
> We need to add metadata into zookeeper about who is the leader for each shard, and have some kind of leader election.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] [Updated] (SOLR-2752) leader-per-shard

Sebastian Nagel (Jira)
In reply to this post by Sebastian Nagel (Jira)

     [ https://issues.apache.org/jira/browse/SOLR-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mark Miller updated SOLR-2752:
------------------------------

    Attachment: SOLR-2752.patch

> leader-per-shard
> ----------------
>
>                 Key: SOLR-2752
>                 URL: https://issues.apache.org/jira/browse/SOLR-2752
>             Project: Solr
>          Issue Type: Sub-task
>          Components: SolrCloud
>            Reporter: Yonik Seeley
>            Assignee: Mark Miller
>             Fix For: 4.0
>
>         Attachments: SOLR-2752.patch
>
>
> We need to add metadata into zookeeper about who is the leader for each shard, and have some kind of leader election.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] [Commented] (SOLR-2752) leader-per-shard

Sebastian Nagel (Jira)
In reply to this post by Sebastian Nagel (Jira)

    [ https://issues.apache.org/jira/browse/SOLR-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13102117#comment-13102117 ]

Mark Miller commented on SOLR-2752:
-----------------------------------

I've hacked out an initial rough patch for this. More tests, refactoring, thinking, etc to come.

Adds a new /collections/{collection}/leader_elect node.

When a core registers, it creates a new ephemeral node under /collections/{collection}/leader_elect/{shard}/election/

eg

/collections/{collection}/leader_elect/{shard}/election/n_0000000001

If that is the lowest n_\*, the core sets itself as the leader in /collections/{collection}/leader_elect/{shard}/leader
If that is not the lowest n_\*, the core puts a watch on the node before it. If that node goes down, the core initiates the leader election process again, see's if it's the lowest n, if so it's the leader, else make a new watch on the n_\* node before it.

Rough early exploration stuff, more to follow.

> leader-per-shard
> ----------------
>
>                 Key: SOLR-2752
>                 URL: https://issues.apache.org/jira/browse/SOLR-2752
>             Project: Solr
>          Issue Type: Sub-task
>          Components: SolrCloud
>            Reporter: Yonik Seeley
>            Assignee: Mark Miller
>             Fix For: 4.0
>
>         Attachments: SOLR-2752.patch
>
>
> We need to add metadata into zookeeper about who is the leader for each shard, and have some kind of leader election.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] [Updated] (SOLR-2752) leader-per-shard

Sebastian Nagel (Jira)
In reply to this post by Sebastian Nagel (Jira)

     [ https://issues.apache.org/jira/browse/SOLR-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mark Miller updated SOLR-2752:
------------------------------

    Attachment: SOLR-2752.patch

new patch - much stronger test, a couple fixes, refactor most of the leader election code into its own class.

> leader-per-shard
> ----------------
>
>                 Key: SOLR-2752
>                 URL: https://issues.apache.org/jira/browse/SOLR-2752
>             Project: Solr
>          Issue Type: Sub-task
>          Components: SolrCloud
>            Reporter: Yonik Seeley
>            Assignee: Mark Miller
>             Fix For: 4.0
>
>         Attachments: SOLR-2752.patch, SOLR-2752.patch
>
>
> We need to add metadata into zookeeper about who is the leader for each shard, and have some kind of leader election.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] [Commented] (SOLR-2752) leader-per-shard

Sebastian Nagel (Jira)
In reply to this post by Sebastian Nagel (Jira)

    [ https://issues.apache.org/jira/browse/SOLR-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13102283#comment-13102283 ]

Mark Miller commented on SOLR-2752:
-----------------------------------

Just a quick correction to first comment - cores create an ephemeral|sequential node - not just ephemeral.

> leader-per-shard
> ----------------
>
>                 Key: SOLR-2752
>                 URL: https://issues.apache.org/jira/browse/SOLR-2752
>             Project: Solr
>          Issue Type: Sub-task
>          Components: SolrCloud
>            Reporter: Yonik Seeley
>            Assignee: Mark Miller
>             Fix For: 4.0
>
>         Attachments: SOLR-2752.patch, SOLR-2752.patch
>
>
> We need to add metadata into zookeeper about who is the leader for each shard, and have some kind of leader election.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] [Updated] (SOLR-2752) leader-per-shard

Sebastian Nagel (Jira)
In reply to this post by Sebastian Nagel (Jira)

     [ https://issues.apache.org/jira/browse/SOLR-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mark Miller updated SOLR-2752:
------------------------------

    Attachment: SOLR-2752.patch

Another new patch:

I moved SolrZooKeeper to the org.apache.zookeeper package so that I could add a simulated timeout method for tests.

I also wrote a new test that starts up a bunch of replicas and then times out the leader. After waiting for the leader to reconnect, all of the other replicas are killed and I check that the first leader is again the leader. I wrote this test because I knew it would fail and that on reconnecting, clients don't jump back into the leader election process.

So I also added to the client reconnection impl - on reconnect, all SolrCores are re-registered. This also has the advantage that any SolrCores that where created while the connection was down are put into play. That allows the new test to pass.

> leader-per-shard
> ----------------
>
>                 Key: SOLR-2752
>                 URL: https://issues.apache.org/jira/browse/SOLR-2752
>             Project: Solr
>          Issue Type: Sub-task
>          Components: SolrCloud
>            Reporter: Yonik Seeley
>            Assignee: Mark Miller
>             Fix For: 4.0
>
>         Attachments: SOLR-2752.patch, SOLR-2752.patch, SOLR-2752.patch
>
>
> We need to add metadata into zookeeper about who is the leader for each shard, and have some kind of leader election.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] [Updated] (SOLR-2752) leader-per-shard

Sebastian Nagel (Jira)
In reply to this post by Sebastian Nagel (Jira)

     [ https://issues.apache.org/jira/browse/SOLR-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mark Miller updated SOLR-2752:
------------------------------

    Attachment: SOLR-2752.patch

feeling motivated I guess - another patch with a bunch of polish

> leader-per-shard
> ----------------
>
>                 Key: SOLR-2752
>                 URL: https://issues.apache.org/jira/browse/SOLR-2752
>             Project: Solr
>          Issue Type: Sub-task
>          Components: SolrCloud
>            Reporter: Yonik Seeley
>            Assignee: Mark Miller
>             Fix For: 4.0
>
>         Attachments: SOLR-2752.patch, SOLR-2752.patch, SOLR-2752.patch, SOLR-2752.patch
>
>
> We need to add metadata into zookeeper about who is the leader for each shard, and have some kind of leader election.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] [Updated] (SOLR-2752) leader-per-shard

Sebastian Nagel (Jira)
In reply to this post by Sebastian Nagel (Jira)

     [ https://issues.apache.org/jira/browse/SOLR-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mark Miller updated SOLR-2752:
------------------------------

    Attachment: SOLR-2752.patch

Another patch I suppose - rename existing tests to LeaderElectionIntegrationTest and a new LeaderElectionTest that just tests the LeaderElector class itself. Also a bit more javadoc.

> leader-per-shard
> ----------------
>
>                 Key: SOLR-2752
>                 URL: https://issues.apache.org/jira/browse/SOLR-2752
>             Project: Solr
>          Issue Type: Sub-task
>          Components: SolrCloud
>            Reporter: Yonik Seeley
>            Assignee: Mark Miller
>             Fix For: 4.0
>
>         Attachments: SOLR-2752.patch, SOLR-2752.patch, SOLR-2752.patch, SOLR-2752.patch, SOLR-2752.patch
>
>
> We need to add metadata into zookeeper about who is the leader for each shard, and have some kind of leader election.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] [Updated] (SOLR-2752) leader-per-shard

Sebastian Nagel (Jira)
In reply to this post by Sebastian Nagel (Jira)

     [ https://issues.apache.org/jira/browse/SOLR-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mark Miller updated SOLR-2752:
------------------------------

    Attachment: SOLR-2752.patch

Another patch I suppose - rename existing tests to LeaderElectionIntegrationTest and a new LeaderElectionTest that just tests the LeaderElector class itself. Also a bit more javadoc.

> leader-per-shard
> ----------------
>
>                 Key: SOLR-2752
>                 URL: https://issues.apache.org/jira/browse/SOLR-2752
>             Project: Solr
>          Issue Type: Sub-task
>          Components: SolrCloud
>            Reporter: Yonik Seeley
>            Assignee: Mark Miller
>             Fix For: 4.0
>
>         Attachments: SOLR-2752.patch, SOLR-2752.patch, SOLR-2752.patch, SOLR-2752.patch, SOLR-2752.patch
>
>
> We need to add metadata into zookeeper about who is the leader for each shard, and have some kind of leader election.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] [Updated] (SOLR-2752) leader-per-shard

Sebastian Nagel (Jira)
In reply to this post by Sebastian Nagel (Jira)

     [ https://issues.apache.org/jira/browse/SOLR-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mark Miller updated SOLR-2752:
------------------------------

    Comment: was deleted

(was: Another patch I suppose - rename existing tests to LeaderElectionIntegrationTest and a new LeaderElectionTest that just tests the LeaderElector class itself. Also a bit more javadoc.)

> leader-per-shard
> ----------------
>
>                 Key: SOLR-2752
>                 URL: https://issues.apache.org/jira/browse/SOLR-2752
>             Project: Solr
>          Issue Type: Sub-task
>          Components: SolrCloud
>            Reporter: Yonik Seeley
>            Assignee: Mark Miller
>             Fix For: 4.0
>
>         Attachments: SOLR-2752.patch, SOLR-2752.patch, SOLR-2752.patch, SOLR-2752.patch, SOLR-2752.patch
>
>
> We need to add metadata into zookeeper about who is the leader for each shard, and have some kind of leader election.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] [Updated] (SOLR-2752) leader-per-shard

Sebastian Nagel (Jira)
In reply to this post by Sebastian Nagel (Jira)

     [ https://issues.apache.org/jira/browse/SOLR-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mark Miller updated SOLR-2752:
------------------------------

    Attachment:     (was: SOLR-2752.patch)

> leader-per-shard
> ----------------
>
>                 Key: SOLR-2752
>                 URL: https://issues.apache.org/jira/browse/SOLR-2752
>             Project: Solr
>          Issue Type: Sub-task
>          Components: SolrCloud
>            Reporter: Yonik Seeley
>            Assignee: Mark Miller
>             Fix For: 4.0
>
>         Attachments: SOLR-2752.patch, SOLR-2752.patch, SOLR-2752.patch, SOLR-2752.patch, SOLR-2752.patch
>
>
> We need to add metadata into zookeeper about who is the leader for each shard, and have some kind of leader election.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] [Commented] (SOLR-2752) leader-per-shard

Sebastian Nagel (Jira)
In reply to this post by Sebastian Nagel (Jira)

    [ https://issues.apache.org/jira/browse/SOLR-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13106622#comment-13106622 ]

Mark Miller commented on SOLR-2752:
-----------------------------------

I think we need a try catch around setting the watch on the next guy in line - he may have been cut down between seeing he was next and setting the watch - we probably want to check if we are the leader again if an exception is thrown setting the watch.

Tried making a stress test that could catch this, but tough window to hit...

> leader-per-shard
> ----------------
>
>                 Key: SOLR-2752
>                 URL: https://issues.apache.org/jira/browse/SOLR-2752
>             Project: Solr
>          Issue Type: Sub-task
>          Components: SolrCloud
>            Reporter: Yonik Seeley
>            Assignee: Mark Miller
>             Fix For: 4.0
>
>         Attachments: SOLR-2752.patch, SOLR-2752.patch, SOLR-2752.patch, SOLR-2752.patch, SOLR-2752.patch
>
>
> We need to add metadata into zookeeper about who is the leader for each shard, and have some kind of leader election.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] [Commented] (SOLR-2752) leader-per-shard

Sebastian Nagel (Jira)
In reply to this post by Sebastian Nagel (Jira)

    [ https://issues.apache.org/jira/browse/SOLR-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13115771#comment-13115771 ]

Mark Miller commented on SOLR-2752:
-----------------------------------

While working on the tests for this I ran into and filed a locale bug in zk: https://issues.apache.org/jira/browse/ZOOKEEPER-1206
               

> leader-per-shard
> ----------------
>
>                 Key: SOLR-2752
>                 URL: https://issues.apache.org/jira/browse/SOLR-2752
>             Project: Solr
>          Issue Type: Sub-task
>          Components: SolrCloud
>            Reporter: Yonik Seeley
>            Assignee: Mark Miller
>             Fix For: 4.0
>
>         Attachments: SOLR-2752.patch, SOLR-2752.patch, SOLR-2752.patch, SOLR-2752.patch, SOLR-2752.patch
>
>
> We need to add metadata into zookeeper about who is the leader for each shard, and have some kind of leader election.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] [Commented] (SOLR-2752) leader-per-shard

Sebastian Nagel (Jira)
In reply to this post by Sebastian Nagel (Jira)

    [ https://issues.apache.org/jira/browse/SOLR-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13116009#comment-13116009 ]

Mark Miller commented on SOLR-2752:
-----------------------------------

I've committed this early work to the solrcloud branch
               

> leader-per-shard
> ----------------
>
>                 Key: SOLR-2752
>                 URL: https://issues.apache.org/jira/browse/SOLR-2752
>             Project: Solr
>          Issue Type: Sub-task
>          Components: SolrCloud
>            Reporter: Yonik Seeley
>            Assignee: Mark Miller
>             Fix For: 4.0
>
>         Attachments: SOLR-2752.patch, SOLR-2752.patch, SOLR-2752.patch, SOLR-2752.patch, SOLR-2752.patch
>
>
> We need to add metadata into zookeeper about who is the leader for each shard, and have some kind of leader election.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] [Resolved] (SOLR-2752) leader-per-shard

Sebastian Nagel (Jira)
In reply to this post by Sebastian Nagel (Jira)

     [ https://issues.apache.org/jira/browse/SOLR-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mark Miller resolved SOLR-2752.
-------------------------------

    Resolution: Fixed
   

> leader-per-shard
> ----------------
>
>                 Key: SOLR-2752
>                 URL: https://issues.apache.org/jira/browse/SOLR-2752
>             Project: Solr
>          Issue Type: Sub-task
>          Components: SolrCloud
>            Reporter: Yonik Seeley
>            Assignee: Mark Miller
>             Fix For: 4.0
>
>         Attachments: SOLR-2752.patch, SOLR-2752.patch, SOLR-2752.patch, SOLR-2752.patch, SOLR-2752.patch
>
>
> We need to add metadata into zookeeper about who is the leader for each shard, and have some kind of leader election.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]