Solr Collection core initialization Error with length mismatch

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Solr Collection core initialization Error with length mismatch

Mohamed Sirajudeen Mayitti Ahamed Pillai
Hello team,

We have 4 Solr VMs in Solr Cloud 7.4. Only a specific node Admin UI, we are seeing the message,
·         cs_signals_shard1_replica_n1: org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: Error opening new searcher

When restarting Solr, noticed below error for the collection in solr.log.

2020-05-26T12:28:29.759-0500 - ERROR [coreContainerWorkExecutor-2-thread-1-processing-n:fzcexfsnbepd02:8983_solr:solr.core.CoreContainer@714] - {node_name=n:fzcexfsnbepd02:8983_solr} - Error waiting for SolrCore to be loaded on startup
org.apache.solr.common.SolrException: Unable to create core [cs_signals_shard1_replica_n1]
                at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1156) ~[solr-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:55:13]
                at org.apache.solr.core.CoreContainer.lambda$load$13(CoreContainer.java:681) ~[solr-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:55:13]
                at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197) ~[metrics-core-3.2.6.jar:3.2.6]
                at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_212]
                at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209) ~[solr-solrj-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:55:14]
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212]
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212]
                at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]
Caused by: org.apache.solr.common.SolrException: Error opening new searcher
                at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1012) ~[solr-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:55:13]
                at org.apache.solr.core.SolrCore.<init>(SolrCore.java:867) ~[solr-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:55:13]
                at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1135) ~[solr-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:55:13]
                ... 7 more
Caused by: org.apache.solr.common.SolrException: Error opening new searcher
                at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2126) ~[solr-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:55:13]
                at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:2246) ~[solr-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:55:13]
                at org.apache.solr.core.SolrCore.initSearcher(SolrCore.java:1095) ~[solr-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:55:13]
                at org.apache.solr.core.SolrCore.<init>(SolrCore.java:984) ~[solr-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:55:13]
                at org.apache.solr.core.SolrCore.<init>(SolrCore.java:867) ~[solr-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:55:13]
                at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1135) ~[solr-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:55:13]
                ... 7 more
Caused by: org.apache.lucene.index.CorruptIndexException: length should be 5271336964 bytes, but is 5272582148 instead (resource=MMapIndexInput(path="/opt/solr-7.4/data/solr/cs_signals_shard1_replica_n1/data/index.20200514060942617/_52b3.cfs"))
                at org.apache.lucene.codecs.lucene50.Lucene50CompoundReader.<init>(Lucene50CompoundReader.java:91) ~[lucene-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:51:45]
                at org.apache.lucene.codecs.lucene50.Lucene50CompoundFormat.getCompoundReader(Lucene50CompoundFormat.java:70) ~[lucene-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:51:45]
                at org.apache.lucene.index.IndexWriter.readFieldInfos(IndexWriter.java:960) ~[lucene-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:51:45]
                at org.apache.lucene.index.IndexWriter.getFieldNumberMap(IndexWriter.java:977) ~[lucene-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:51:45]
                at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:869) ~[lucene-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:51:45]
                at org.apache.solr.update.SolrIndexWriter.<init>(SolrIndexWriter.java:124) ~[solr-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:55:13]
                at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:97) ~[solr-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:55:13]
                at org.apache.solr.update.DefaultSolrCoreState.createMainIndexWriter(DefaultSolrCoreState.java:257) ~[solr-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:55:13]
                at org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:131) ~[solr-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:55:13]
                at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2087) ~[solr-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:55:13]
                at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:2246) ~[solr-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:55:13]
                at org.apache.solr.core.SolrCore.initSearcher(SolrCore.java:1095) ~[solr-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:55:13]
                at org.apache.solr.core.SolrCore.<init>(SolrCore.java:984) ~[solr-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:55:13]
                at org.apache.solr.core.SolrCore.<init>(SolrCore.java:867) ~[solr-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:55:13]
                at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1135) ~[solr-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:55:13]
                ... 7 more


How do we recover from this error? This collection has 1 shard and 2 replicas, the error noticed is on the down replica, we are currently running with other replica which is active.

Thanks,
Mohamed