Solr 8.2.0 - Unable to write response

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Solr 8.2.0 - Unable to write response

Joe Obernberger
Hi All - getting this error from only one server in a 45 node cluster
when calling COLSTATUS.  Any ideas?


2019-11-01 13:17:32.556 INFO  (qtp694316372-44709) [   ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections
params={name=UNCLASS_2019_1_18_36&action=COLSTATUS&wt=javabin&version=2}
status=0 QTime=94734
2019-11-01 13:17:32.567 INFO  (qtp694316372-44688) [   ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections
params={name=UNCLASS_2021_2_17_36&action=COLSTATUS&wt=javabin&version=2}
status=0 QTime=815338
2019-11-01 13:17:32.570 INFO  (qtp694316372-44688) [   ]
o.a.s.s.HttpSolrCall Unable to write response, client closed connection
or we are shutting down => org.eclipse.jetty.io.EofException: Closed
         at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:491)
org.eclipse.jetty.io.EofException: Closed
         at
org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:491)
~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
         at
org.apache.solr.servlet.ServletOutputStreamWrapper.write(ServletOutputStreamWrapper.java:134)
~[?:?]
         at
org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:216)
~[?:?]
         at
org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:209)
~[?:?]
         at
org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:170)
~[?:?]
         at
org.apache.solr.response.BinaryResponseWriter.write(BinaryResponseWriter.java:60)
~[?:?]
         at
org.apache.solr.response.QueryResponseWriterUtil.writeQueryResponse(QueryResponseWriterUtil.java:49)
~[?:?]
         at
org.apache.solr.servlet.HttpSolrCall.writeResponse(HttpSolrCall.java:873)
~[?:?]
         at
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:793)
~[?:?]
         at
org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:546) ~[?:?]
         at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:423)
~[?:?]
         at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:350)
~[?:?]
         at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
~[jetty-servlet-9.4.19.v20190610.jar:9.4.19.v20190610]
         at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
~[jetty-servlet-9.4.19.v20190610.jar:9.4.19.v20190610]
         at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
         at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
~[jetty-security-9.4.19.v20190610.jar:9.4.19.v20190610]
         at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
         at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
         at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1711)
~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
         at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
         at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1347)
~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
         at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
         at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
~[jetty-servlet-9.4.19.v20190610.jar:9.4.19.v20190610]
         at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1678)
~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
         at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
         at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1249)
~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
         at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
         at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
         at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:152)
~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
         at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
         at
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
~[jetty-rewrite-9.4.19.v20190610.jar:9.4.19.v20190610]
         at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
         at org.eclipse.jetty.server.Server.handle(Server.java:505)
~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
         at
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370)
~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
         at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267)
~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
         at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
~[jetty-io-9.4.19.v20190610.jar:9.4.19.v20190610]
         at
org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
~[jetty-io-9.4.19.v20190610.jar:9.4.19.v20190610]
         at
org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
~[jetty-io-9.4.19.v20190610.jar:9.4.19.v20190610]
         at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
~[jetty-util-9.4.19.v20190610.jar:9.4.19.v20190610]
         at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
~[jetty-util-9.4.19.v20190610.jar:9.4.19.v20190610]
         at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
~[jetty-util-9.4.19.v20190610.jar:9.4.19.v20190610]
         at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
~[jetty-util-9.4.19.v20190610.jar:9.4.19.v20190610]
         at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
~[jetty-util-9.4.19.v20190610.jar:9.4.19.v20190610]
         at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:781)
~[jetty-util-9.4.19.v20190610.jar:9.4.19.v20190610]
         at
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:917)
~[jetty-util-9.4.19.v20190610.jar:9.4.19.v20190610]
         at java.lang.Thread.run(Thread.java:748) [?:1.8.0_232]
         Suppressed: org.eclipse.jetty.io.EofException: Closed
                 at
org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:491)
~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
                 at
org.apache.solr.servlet.ServletOutputStreamWrapper.write(ServletOutputStreamWrapper.java:134)
~[?:?]
                 at
org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:216)
~[?:?]
                 at
org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:209)
~[?:?]
                 at
org.apache.solr.common.util.JavaBinCodec.close(JavaBinCodec.java:1262)
~[?:?]
                 at
org.apache.solr.response.BinaryResponseWriter.write(BinaryResponseWriter.java:61)
~[?:?]
                 at
org.apache.solr.response.QueryResponseWriterUtil.writeQueryResponse(QueryResponseWriterUtil.java:49)
~[?:?]
                 at
org.apache.solr.servlet.HttpSolrCall.writeResponse(HttpSolrCall.java:873)
~[?:?]
                 at
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:793)
~[?:?]
                 at
org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:546) ~[?:?]
                 at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:423)
~[?:?]
                 at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:350)
~[?:?]
                 at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
~[jetty-servlet-9.4.19.v20190610.jar:9.4.19.v20190610]
                 at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
~[jetty-servlet-9.4.19.v20190610.jar:9.4.19.v20190610]
                 at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
                 at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
~[jetty-security-9.4.19.v20190610.jar:9.4.19.v20190610]
                 at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
                 at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
                 at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1711)
~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
                 at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
                 at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1347)
~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
                 at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
                 at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
~[jetty-servlet-9.4.19.v20190610.jar:9.4.19.v20190610]
                 at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1678)
~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
                 at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
                 at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1249)
~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]

-Joe

Reply | Threaded
Open this post in threaded view
|

Re: Solr 8.2.0 - Unable to write response

Shawn Heisey-2
On 11/1/2019 7:20 AM, Joe Obernberger wrote:
> Hi All - getting this error from only one server in a 45 node cluster
> when calling COLSTATUS.  Any ideas?

<snip>

> 2019-11-01 13:17:32.556 INFO  (qtp694316372-44709) [   ]
> o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections
> params={name=UNCLASS_2019_1_18_36&action=COLSTATUS&wt=javabin&version=2}
> status=0 QTime=94734
> 2019-11-01 13:17:32.567 INFO  (qtp694316372-44688) [   ]
> o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections
> params={name=UNCLASS_2021_2_17_36&action=COLSTATUS&wt=javabin&version=2}
> status=0 QTime=815338
> 2019-11-01 13:17:32.570 INFO  (qtp694316372-44688) [   ]
> o.a.s.s.HttpSolrCall Unable to write response, client closed connection
> or we are shutting down => org.eclipse.jetty.io.EofException: Closed
>          at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:491)
> org.eclipse.jetty.io.EofException: Closed

Jetty's EofException almost always means one specific thing.  The client
closed the connection before Solr could respond, so when Solr finally
finished processing and tried to have Jetty send the response, there was
nowhere to send it -- the connection was gone.

The first two lines of the log snippet indicate that there was one
COLSTATUS call that took nearly 95 seconds, and one that took 815
seconds, which is close to 15 minutes.  Apparently those two calls
completed at the same time, even though they did not start at the same
time.  Which suggests that for many minutes, the Solr server has been
under severe stress that prevented it from responding quickly to the
COLSTATUS request.  The client gave up and closed its connection ...
probably on the one that took almost 15 minutes.

I found that the COLSTATUS action is mentioned in the 8.1 reference
guide on the "Collections API" page, but it is not there on the same
page in the 8.2 guide.  That page in 8.2 appears to be significantly
smaller and missing most of the documentation that 8.1 has.  So I think
we had a problem with documentation generation on 8.2.

Based on what I can see in the response on the 8.1 guide, I'm betting
that gathering COLSTATUS takes a fair amount of processing, and any
performance issues will make it very slow.

Thanks,
Shawn
Reply | Threaded
Open this post in threaded view
|

Re: Solr 8.2.0 - Unable to write response

Joe Obernberger
Thank you Shawn.

What I'm trying to get for my application is the commitTimeMSec. I use
that value to build up an alias of solr collections.  Is there a better way?

-Joe

On 11/1/2019 10:17 AM, Shawn Heisey wrote:

> On 11/1/2019 7:20 AM, Joe Obernberger wrote:
>> Hi All - getting this error from only one server in a 45 node cluster
>> when calling COLSTATUS.  Any ideas?
>
> <snip>
>
>> 2019-11-01 13:17:32.556 INFO (qtp694316372-44709) [   ]
>> o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections
>> params={name=UNCLASS_2019_1_18_36&action=COLSTATUS&wt=javabin&version=2}
>> status=0 QTime=94734
>> 2019-11-01 13:17:32.567 INFO  (qtp694316372-44688) [   ]
>> o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections
>> params={name=UNCLASS_2021_2_17_36&action=COLSTATUS&wt=javabin&version=2}
>> status=0 QTime=815338
>> 2019-11-01 13:17:32.570 INFO  (qtp694316372-44688) [   ]
>> o.a.s.s.HttpSolrCall Unable to write response, client closed
>> connection or we are shutting down =>
>> org.eclipse.jetty.io.EofException: Closed
>>          at
>> org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:491)
>> org.eclipse.jetty.io.EofException: Closed
>
> Jetty's EofException almost always means one specific thing.  The
> client closed the connection before Solr could respond, so when Solr
> finally finished processing and tried to have Jetty send the response,
> there was nowhere to send it -- the connection was gone.
>
> The first two lines of the log snippet indicate that there was one
> COLSTATUS call that took nearly 95 seconds, and one that took 815
> seconds, which is close to 15 minutes.  Apparently those two calls
> completed at the same time, even though they did not start at the same
> time.  Which suggests that for many minutes, the Solr server has been
> under severe stress that prevented it from responding quickly to the
> COLSTATUS request.  The client gave up and closed its connection ...
> probably on the one that took almost 15 minutes.
>
> I found that the COLSTATUS action is mentioned in the 8.1 reference
> guide on the "Collections API" page, but it is not there on the same
> page in the 8.2 guide.  That page in 8.2 appears to be significantly
> smaller and missing most of the documentation that 8.1 has.  So I
> think we had a problem with documentation generation on 8.2.
>
> Based on what I can see in the response on the 8.1 guide, I'm betting
> that gathering COLSTATUS takes a fair amount of processing, and any
> performance issues will make it very slow.
>
> Thanks,
> Shawn
>