Auto commit Error - Solr Cloud 6.6.0 with HDFS

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Auto commit Error - Solr Cloud 6.6.0 with HDFS

Joe Obernberger
Started up a 6.6.0 solr cloud instance running on 45 machines yesterday
using HDFS (managed schema in zookeeper) and began indexing.  This error
occurred on several of the nodes:

auto commit error...:org.apache.solr.common.SolrException: Error opening
new searcher
     at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2069)
     at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:2189)
     at
org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:667)
     at org.apache.solr.update.CommitTracker.run(CommitTracker.java:217)
     at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
     at java.util.concurrent.FutureTask.run(FutureTask.java:266)
     at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
     at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
     at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
     at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
     at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.solr.common.SolrException: openNewSearcher called
on closed core
     at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2056)
     ... 10 more
7/12/2017, 8:53:53 AM

Immediately following this error, the log shows another:

auto commit error...:org.apache.solr.common.SolrException:
openNewSearcher called on closed core
     at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1943)
     at
org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:678)
     at org.apache.solr.update.CommitTracker.run(CommitTracker.java:217)
     at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
     at java.util.concurrent.FutureTask.run(FutureTask.java:266)
     at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
     at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
     at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
     at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
     at java.lang.Thread.run(Thread.java:748)

Any ideas on what the problem could be?  Thank you!

-Joe

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Auto commit Error - Solr Cloud 6.6.0 with HDFS

Shawn Heisey-2
On 7/12/2017 7:14 AM, Joe Obernberger wrote:
> Started up a 6.6.0 solr cloud instance running on 45 machines
> yesterday using HDFS (managed schema in zookeeper) and began
> indexing.  This error occurred on several of the nodes:
<snip>
> Caused by: org.apache.solr.common.SolrException: openNewSearcher
> called on closed core

There's the important part of the error.

For some reason, which is not immediately clear from the information
provided, the core is closed.  In that situation, Solr is not able to
open a new searcher, so this error happens.  Do you have any other WARN
or ERROR messages in solr.log before this error?  You might want to find
a way to share an entire logfile and provide a URL for accessing it.

Thanks,
Shawn

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Auto commit Error - Solr Cloud 6.6.0 with HDFS

Joe Obernberger
Thank you Shawn - we have some WARN messages like :

DFSClient
Slow waitForAckedSeqno took 31066ms (threshold=30000ms)

and

DFSClient
Slow ReadProcessor read fields took 30737ms (threshold=30000ms); ack:
seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS
downstreamAckTimeNanos: 30735948057,&#8203; targets:
[DatanodeInfoWithStorage[172.16.100.223:50010,&#8203;DS-fc197ece-4c43-46a4-964e-9ecc8adcdcb1,&#8203;DISK],&#8203;
DatanodeInfoWithStorage[172.16.100.228:50010,&#8203;DS-fd9fe2ea-780b-490a-b9a7-ad54c994e307,&#8203;DISK],&#8203;
DatanodeInfoWithStorage[172.16.20.17:50010,&#8203;DS-63b38ca5-b674-4cdc-8102-f07a22709dc1,&#8203;DISK]]

I was getting the log ready for you, but it was overwritten in the
interim.  If it happens again, I'll get the log file ready.

-Joe


On 7/12/2017 9:25 AM, Shawn Heisey wrote:

> On 7/12/2017 7:14 AM, Joe Obernberger wrote:
>> Started up a 6.6.0 solr cloud instance running on 45 machines
>> yesterday using HDFS (managed schema in zookeeper) and began
>> indexing.  This error occurred on several of the nodes:
> <snip>
>> Caused by: org.apache.solr.common.SolrException: openNewSearcher
>> called on closed core
> There's the important part of the error.
>
> For some reason, which is not immediately clear from the information
> provided, the core is closed.  In that situation, Solr is not able to
> open a new searcher, so this error happens.  Do you have any other WARN
> or ERROR messages in solr.log before this error?  You might want to find
> a way to share an entire logfile and provide a URL for accessing it.
>
> Thanks,
> Shawn
>
>
> ---
> This email has been checked for viruses by AVG.
> http://www.avg.com
>

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Auto commit Error - Solr Cloud 6.6.0 with HDFS

Joe Obernberger
In reply to this post by Shawn Heisey-2
Hi Shawn - had a shard go down (appears to have just dropped out of the
cluster) with a similar error:

2017-07-14 20:43:04.238 ERROR (commitScheduler-65-thread-1) [c:UNCLASS
s:shard87 r:core_node132 x:UNCLASS_shard87_replica1]
o.a.s.u.CommitTracker auto commit
error...:org.apache.solr.common.SolrException: openNewSearcher called on
closed core
         at
org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1943)
         at
org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:678)
         at org.apache.solr.update.CommitTracker.run(CommitTracker.java:217)
         at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
         at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
         at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
         at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
         at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
         at java.lang.Thread.run(Thread.java:748)

The whole log can be found here:
http://lovehorsepower.com/solr.log
the GC log is here:
http://lovehorsepower.com/solr_gc.log.3.current

-Joe


On 7/12/2017 9:25 AM, Shawn Heisey wrote:

> On 7/12/2017 7:14 AM, Joe Obernberger wrote:
>> Started up a 6.6.0 solr cloud instance running on 45 machines
>> yesterday using HDFS (managed schema in zookeeper) and began
>> indexing.  This error occurred on several of the nodes:
> <snip>
>> Caused by: org.apache.solr.common.SolrException: openNewSearcher
>> called on closed core
> There's the important part of the error.
>
> For some reason, which is not immediately clear from the information
> provided, the core is closed.  In that situation, Solr is not able to
> open a new searcher, so this error happens.  Do you have any other WARN
> or ERROR messages in solr.log before this error?  You might want to find
> a way to share an entire logfile and provide a URL for accessing it.
>
> Thanks,
> Shawn
>
>
> ---
> This email has been checked for viruses by AVG.
> http://www.avg.com
>

Loading...