Solr waitForMerges() causing leaderless shard during shutdown

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Solr waitForMerges() causing leaderless shard during shutdown

Ramsey Haddad (BLOOMBERG/ LONDON)
Hi Solr community,

We have been investigating an issue in our solr (7.5.0) setup where the shutdown of our solr node takes quite some time (3-4 minutes) during which we are effectively leaderless.
After investigating and digging deeper we were able to track it down to segment merges which happen before a solr core is closed.

************************************ stack trace when killing the node ************************************

Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.131-b11 mixed mode):

"Attach Listener" #150736 daemon prio=9 os_prio=0 tid=0x00007f6da4002000 nid=0x13292 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"coreCloseExecutor-22-thread-1" #150733 prio=5 os_prio=0 tid=0x00007f6d54020800 nid=0x11b61 in Object.wait() [0x00007f6c98564000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
~at java.lang.Object.wait(Native Method)
~at org.apache.lucene.index.IndexWriter.doWait(IndexWriter.java:4672)
~- locked <0x00000005499908c0> (a org.apache.solr.update.SolrIndexWriter)
~at org.apache.lucene.index.IndexWriter.waitForMerges(IndexWriter.java:2559)
~- locked <0x00000005499908c0> (a org.apache.solr.update.SolrIndexWriter)
~at org.apache.lucene.index.IndexWriter.shutdown(IndexWriter.java:1036)
~at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1078)
~at org.apache.solr.update.SolrIndexWriter.close(SolrIndexWriter.java:286)
~at org.apache.solr.update.DirectUpdateHandler2.closeWriter(DirectUpdateHandler2.java:892)
~at org.apache.solr.update.DefaultSolrCoreState.closeIndexWriter(DefaultSolrCoreState.java:105)
~at org.apache.solr.update.DefaultSolrCoreState.close(DefaultSolrCoreState.java:399)
~- locked <0x000000054e150cc0> (a org.apache.solr.update.DefaultSolrCoreState)
~at org.apache.solr.update.SolrCoreState.decrefSolrCoreState(SolrCoreState.java:83)
~at org.apache.solr.core.SolrCore.close(SolrCore.java:1574)
~at org.apache.solr.core.SolrCores.lambda$close$0(SolrCores.java:141)
~at org.apache.solr.core.SolrCores$$Lambda$443/1058423472.call(Unknown Source)
~at java.util.concurrent.FutureTask.run(FutureTask.java:266)

************************************************************************************************************


The situation is as follows -

1. The first thing that happens is the request handlers being closed at -
https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/core/SolrCore.java#L1588

2. Then it tries to close the index writer via -
https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/core/SolrCore.java#L1610

3. When closing the index writer, it waits for any pending merges to finish at -
https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/IndexWriter.java#L1236

Now, if this waitForMerges() takes a long time (3-4 minutes), the instance won't shut down for the whole of that time, but because of *Step 1* it will stop
accepting any requests.

This becomes a problem when this node has a leader replica and it is stuck on waitForMerges() after closing its reqHandlers. We are in a situation where
the leader is not accepting requests but has not given away the leadership, so we are in a leaderless phase.


This issue triggers when we turnaround our nodes which causes a brief period of leaderless shards which leads to potential data losses.

My question is -
1. How to avoid this situation given that we have big segment sizes and the merging the largest segments is going to take some time.
We do not want to reduce the segment size as it will impact our search performance which is crucial.
2. Should Solr ideally not do the waitForMerges() step before closing the request handlers?


Merge Policy config and segment size -

<mergePolicyFactory class="org.apache.solr.index.SortingMergePolicyFactory">
<str name="sort">time_of_arrival desc</str>
<str name="wrapped.prefix">inner</str>
<str name="inner.class">org.apache.solr.index.TieredMergePolicyFactory</str>
<!-- Override default Solr7 values with the values
we have been using in Solr 4, which allow
more segments to be merged at once and larger
segments to be created.
-->
<int name="inner.maxMergeAtOnce">16</int>
<int name="inner.maxMergedSegmentMB">20480</int>
</mergePolicyFactory>