[jira] [Commented] (SOLR-10265) Overseer can become the bottleneck in very large clusters

Previous Topic Next Topic
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
Report Content as Inappropriate

[jira] [Commented] (SOLR-10265) Overseer can become the bottleneck in very large clusters

JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/SOLR-10265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15931532#comment-15931532 ]

Shawn Heisey commented on SOLR-10265:

bq. We should seriously reconsider the idea of using ZK queue for the overseer messages.

Serious question, not being confrontational:  What are the characteristics of zookeeper which make it bad choice for the overseer queue?  My suspicion is that it's problematic because of the number of messages in a typical queue, not because of technology issues.  If a way can be found to accomplish the overseer job with a very small number of messages instead of a very large number, would ZK be suitable?

Can't we create more queue entry types that can accomplish with a single entry what currently can take hundreds or thousands of entries (depending on the number of collections)?  I can envision types that indicate "node X just came up" and "node X just went down" ... the overseer should be able to use the cluster information in ZK to decide what state changes are required, rather than processing an individual message for each state change.  Other types that I haven't thought of would probably be needed.  Another type might specify state changes that affect multiple collections or shard replicas with a single entry.

When a collection is created, is there REALLY a need to send state changes for EVERY collection in the cloud?  That seems completely unnecessary to me.  I have not attempted other actions like creating a new replica, but I would not be surprised to learn that they also send many unnecessary state changes.

If ZK isn't used for the overseer queue, then some other method must be found to make sure the queue is not lost when nodes go down.  An in-memory queue would need to be somehow replicated to at least one other node in the cluster.  Doing that well probably means finding another dependency where somebody else has worked out the bugs.

> Overseer can become the bottleneck in very large clusters
> ---------------------------------------------------------
>                 Key: SOLR-10265
>                 URL: https://issues.apache.org/jira/browse/SOLR-10265
>             Project: Solr
>          Issue Type: Improvement
>      Security Level: Public(Default Security Level. Issues are Public)
>            Reporter: Varun Thacker
> Let's say we have a large cluster. Some numbers:
> - To ingest the data at the volume we want to I need roughly a 600 shard collection.
> - Index into the collection for 1 hour and then create a new collection
> - For a 30 days retention window with these numbers we would end up wth  ~400k cores in the cluster
> - Just a rolling restart of this cluster can take hours because the overseer queue gets backed up. If a few nodes looses connectivity to ZooKeeper then also we can end up with lots of messages in the Overseer queue
> With some tests here are the two high level problems we have identified:
> 1> How fast can the overseer process operations:
> The rate at which the overseer processes events is too slow at this scale.
> I ran {{OverseerTest#testPerformance}} which creates 10 collections ( 1 shard 1 replica ) and generates 20k state change events. The test took 119 seconds to run on my machine which means ~170 events a second. Let's say a server can process 5x of my machine so 1k events a second.
> Total events generated by a 400k replica cluster = 400k * 4 ( state changes till replica become active ) = 1.6M / 1k events a second will be 1600 minutes.
> Second observation was that the rate at which the overseer can process events slows down when the number of items in the queue gets larger
> I ran the same {{OverseerTest#testPerformance}} but changed the number of events generated to 2000 instead. The test took only 5 seconds to run. So it was a lot faster than the test run which generated 20k events
> 2> State changes overwhelming ZK:
> For every state change Solr is writing out a big state.json to zookeeper. This can lead to the zookeeper transaction logs going out of control even with auto purging etc set .
> I haven't debugged why the transaction logs ran into terabytes without taking into snapshots but this was my assumption based on the other problems we observed

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]