Zk Status Error

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Zk Status Error

Sadiki Latty

Hi,

 

I am in the process of upgrading my solr cloud from the 7.2.1 to the latest 7.7.1 and it seems to have successfully installed, my data is still there, still searchable and the other tabs are still working under the Cloud section, except the ZK Status section. I am seeing an error when I try to  view the “ZK Status” tab in the Solr Admin UI.

 

https://pasteboard.co/Ibv53co.png

 

 

 

Here are the 2 errors in the Solr Logging section

 

RequestHandlerBase  java.lang.ArrayIndexOutOfBoundsException: 1

“java.lang.ArrayIndexOutOfBoundsException: 1

                at org.apache.solr.handler.admin.ZookeeperStatusHandler.monitorZookeeper(ZookeeperStatusHandler.java:185)

                at org.apache.solr.handler.admin.ZookeeperStatusHandler.getZkStatus(ZookeeperStatusHandler.java:99)

                at org.apache.solr.handler.admin.ZookeeperStatusHandler.handleRequestBody(ZookeeperStatusHandler.java:77)

                at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)

                at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:735)

                at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:716)

                at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:496)

                at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:395)

                at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:341)

                at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)

                at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)

                at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)

                at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)

                at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)

                at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)

                at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)

                at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)

                at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)

                at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)

                at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)

                at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)

                at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)

                at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)

                at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)

                at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)

                at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)

                at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)

                at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)

                at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)

                at org.eclipse.jetty.server.Server.handle(Server.java:502)

                at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)

                at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)

                at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)

                at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)

                at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)

                at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)

                at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)

                at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)

                at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)

                at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)

                at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)

                at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)

                at java.lang.Thread.run(Thread.java:748)

 

 

Error #2: 

HttpSolrCall null:java.lang.ArrayIndexOutOfBoundsException: 1

 

null:java.lang.ArrayIndexOutOfBoundsException: 1

                at org.apache.solr.handler.admin.ZookeeperStatusHandler.monitorZookeeper(ZookeeperStatusHandler.java:185)

                at org.apache.solr.handler.admin.ZookeeperStatusHandler.getZkStatus(ZookeeperStatusHandler.java:99)

                at org.apache.solr.handler.admin.ZookeeperStatusHandler.handleRequestBody(ZookeeperStatusHandler.java:77)

                at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)

                at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:735)

                at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:716)

                at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:496)

                at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:395)

                at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:341)

                at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)

                at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)

                at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)

                at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)

                at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)

                at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)

                at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)

                at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)

                at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)

                at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)

                at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)

                at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)

                at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)

                at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)

                at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)

                at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)

                at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)

                at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)

                at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)

                at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)

                at org.eclipse.jetty.server.Server.handle(Server.java:502)

                at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)

                at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)

                at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)

                at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)

                at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)

                at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)

                at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)

                at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)

                at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)

                at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)

                at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)

                at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)

                at java.lang.Thread.run(Thread.java:748)

 

I also tried looking at the log file on the server (solr.log) but I am not seeing these errors in there.

 

Any other steps to try would be greatly appreciated.

 

Regards,

Sid

 

Reply | Threaded
Open this post in threaded view
|

Re: Zk Status Error

Shawn Heisey-2
On 4/23/2019 12:14 PM, Sadiki Latty wrote:
> Here are the 2 errors in the Solr Logging section
>
> RequestHandlerBase  java.lang.ArrayIndexOutOfBoundsException: 1
>
> “java.lang.ArrayIndexOutOfBoundsException: 1
>
>                  at
> org.apache.solr.handler.admin.ZookeeperStatusHandler.monitorZookeeper(ZookeeperStatusHandler.java:185)

That line of code in Solr 7.7.1 is:

       obj.put(line.split("\t")[0], line.split("\t")[1]);

The error is saying that the array created by the split function didn't
have two elements, so asking for the entry numbered 1 doesn't work.

Which means that the output Solr received from the ZK "mntr" command was
not what Solr expected.

What version of ZK do you have running your ZK ensemble?  Is your ZK
ensemble working correctly?  The ZK client version in Solr 7.7.1 is
3.4.13.  If your server version is different, maybe there was a change
in the output from the mntr command.

What happens if you issue the mntr command yourself directly?

Solr should, at the very least, detect possible problems with decoding
the response from ZK and display a more helpful message.  If there was a
change in the command output, Solr should be resilient enough to handle it.

Thanks,
Shawn
Reply | Threaded
Open this post in threaded view
|

RE: Zk Status Error

Sadiki Latty
Hey Shawn,

Thanks for the response. I was using 3.5.3-beta and after your suggestion I tried 3.5.4-beta as well as the latest 3.4.* Zookeeper with the same issue.

Trying to execute the mntr command directly revealed the problem because I got an error message that said that the mntr command was not whitelisted. Once I added the mntr command to the whitelist everything is up and functional.

Thanks for the guidance.

Sadiki Latty

-----Original Message-----
From: Shawn Heisey [mailto:[hidden email]]
Sent: April 23, 2019 3:58 PM
To: [hidden email]
Subject: Re: Zk Status Error

On 4/23/2019 12:14 PM, Sadiki Latty wrote:
> Here are the 2 errors in the Solr Logging section
>
> RequestHandlerBase  java.lang.ArrayIndexOutOfBoundsException: 1
>
> "java.lang.ArrayIndexOutOfBoundsException: 1
>
>                  at
> org.apache.solr.handler.admin.ZookeeperStatusHandler.monitorZookeeper(
> ZookeeperStatusHandler.java:185)

That line of code in Solr 7.7.1 is:

       obj.put(line.split("\t")[0], line.split("\t")[1]);

The error is saying that the array created by the split function didn't have two elements, so asking for the entry numbered 1 doesn't work.

Which means that the output Solr received from the ZK "mntr" command was not what Solr expected.

What version of ZK do you have running your ZK ensemble?  Is your ZK ensemble working correctly?  The ZK client version in Solr 7.7.1 is 3.4.13.  If your server version is different, maybe there was a change in the output from the mntr command.

What happens if you issue the mntr command yourself directly?

Solr should, at the very least, detect possible problems with decoding the response from ZK and display a more helpful message.  If there was a change in the command output, Solr should be resilient enough to handle it.

Thanks,
Shawn
Reply | Threaded
Open this post in threaded view
|

RE: Zk Status Error

mdsholund
I am also getting this error using ZK 3.5.5 and Solr 7.7.2.  I have
whitelisted mntr but still get a similar exception

2019-10-10 14:59:01.799 ERROR (qtp591391158-152) [   ] o.a.s.s.HttpSolrCall
null:java.lang.ArrayIndexOutOfBoundsException: 1
        at
org.apache.solr.handler.admin.ZookeeperStatusHandler.monitorZookeeper(ZookeeperStatusHandler.java:189)


As far as I know my ensemble is working fine.  The output of the mntr
command looks like it is all at least two values.  Is there a way that I can
see what it is choking on?



--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Reply | Threaded
Open this post in threaded view
|

Re: Zk Status Error

Shawn Heisey-2
On 10/10/2019 9:00 AM, mdsholund wrote:

> I am also getting this error using ZK 3.5.5 and Solr 7.7.2.  I have
> whitelisted mntr but still get a similar exception
>
> 2019-10-10 14:59:01.799 ERROR (qtp591391158-152) [   ] o.a.s.s.HttpSolrCall
> null:java.lang.ArrayIndexOutOfBoundsException: 1
>          at
> org.apache.solr.handler.admin.ZookeeperStatusHandler.monitorZookeeper(ZookeeperStatusHandler.java:189)
>
> As far as I know my ensemble is working fine.  The output of the mntr
> command looks like it is all at least two values.  Is there a way that I can
> see what it is choking on?

You may be running into this problem:

https://issues.apache.org/jira/browse/SOLR-13672

ZK 3.5 changed the output of the "conf" 4lw command in a way that is
incompatible with Solr code.  We consider the problem to be a ZK bug,
but worked around it in Solr because ZK's typical release schedule is
very slow.  A new Solr release will come out long before a new ZK release.

Even Solr 8.2.0, which was updated to the ZK 3.5.5 client, has this
problem.  It will be fixed in 8.3.0 when that version is released.

Thanks,
Shawn