Exception when processing streaming expression

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Exception when processing streaming expression

Christian Spitzlay
Hi,

I am seeing a lot of (reproducible) exceptions in my solr log file
when I execute streaming expressions:

o.a.s.s.HttpSolrCall  Unable to write response, client closed connection or we are shutting down
org.eclipse.jetty.io.EofException
        at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:292)
        at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:429)
        at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:322)
        at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:372)
        at org.eclipse.jetty.server.HttpConnection$SendCallback.process(HttpConnection.java:794)
[…]
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
        at java.base/java.lang.Thread.run(Thread.java:844)
Caused by: java.io.IOException: Broken pipe
        at java.base/sun.nio.ch.FileDispatcherImpl.writev0(Native Method)
        at java.base/sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:51)
        at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:148)
        at java.base/sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:506)
        at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:272)
        ... 69 more


I have read up on the exception message and found
http://lucene.472066.n3.nabble.com/Unable-to-write-response-client-closed-connection-or-we-are-shutting-down-tt4350349.html#a4350947
but I don’t understand how an early client connect can cause what I am seeing:

What puzzles me is that the response has been delivered in full to the
client library, including the document with EOF.

So Solr must have already processed the streaming expression and returned the result.
It’s just that the log is filled with stacktraces of this exception that suggests something went wrong.
I don’t understand why this happens when the query seems to have succeeded.


Best regards,
Christian


Reply | Threaded
Open this post in threaded view
|

Re: Exception when processing streaming expression

Joel Bernstein
Can your provide some example expressions that are causing these exceptions?

Joel Bernstein
http://joelsolr.blogspot.com/

On Wed, Jun 13, 2018 at 9:02 AM, Christian Spitzlay <
[hidden email]> wrote:

> Hi,
>
> I am seeing a lot of (reproducible) exceptions in my solr log file
> when I execute streaming expressions:
>
> o.a.s.s.HttpSolrCall  Unable to write response, client closed connection
> or we are shutting down
> org.eclipse.jetty.io.EofException
>         at org.eclipse.jetty.io.ChannelEndPoint.flush(
> ChannelEndPoint.java:292)
>         at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:429)
>         at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:322)
>         at org.eclipse.jetty.io.AbstractEndPoint.write(
> AbstractEndPoint.java:372)
>         at org.eclipse.jetty.server.HttpConnection$SendCallback.
> process(HttpConnection.java:794)
> […]
>         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(
> EatWhatYouKill.java:131)
>         at org.eclipse.jetty.util.thread.ReservedThreadExecutor$
> ReservedThread.run(ReservedThreadExecutor.java:382)
>         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
> QueuedThreadPool.java:708)
>         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(
> QueuedThreadPool.java:626)
>         at java.base/java.lang.Thread.run(Thread.java:844)
> Caused by: java.io.IOException: Broken pipe
>         at java.base/sun.nio.ch.FileDispatcherImpl.writev0(Native Method)
>         at java.base/sun.nio.ch.SocketDispatcher.writev(
> SocketDispatcher.java:51)
>         at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:148)
>         at java.base/sun.nio.ch.SocketChannelImpl.write(
> SocketChannelImpl.java:506)
>         at org.eclipse.jetty.io.ChannelEndPoint.flush(
> ChannelEndPoint.java:272)
>         ... 69 more
>
>
> I have read up on the exception message and found
> http://lucene.472066.n3.nabble.com/Unable-to-write-response-client-closed-
> connection-or-we-are-shutting-down-tt4350349.html#a4350947
> but I don’t understand how an early client connect can cause what I am
> seeing:
>
> What puzzles me is that the response has been delivered in full to the
> client library, including the document with EOF.
>
> So Solr must have already processed the streaming expression and returned
> the result.
> It’s just that the log is filled with stacktraces of this exception that
> suggests something went wrong.
> I don’t understand why this happens when the query seems to have succeeded.
>
>
> Best regards,
> Christian
>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Exception when processing streaming expression

Christian Spitzlay
Here ist one I stripped down as far as I could:

innerJoin(sort(search(kmm, q="sds_endpoint_uuid:(2f927a0b\-fe38\-451e\-9103\-580914a77e82)", fl="sds_endpoint_uuid,sds_to_endpoint_uuid", sort="sds_to_endpoint_uuid ASC", qt="/export"), by="sds_endpoint_uuid ASC"), search(kmm, q=ss_search_api_datasource:entity\:as_metadata, fl="sds_metadata_of_uuid", sort="sds_metadata_of_uuid ASC", qt="/select", rows=10000), on="sds_endpoint_uuid=sds_metadata_of_uuid")

The exception happens both via PHP (search_api_solr / Solarium) and via the Solr admin UI.
(version: Solr 7.3.1 on macOS High Sierra 10.13.5)

It seems to be related to the fact that the second stream uses "select“.
- If I use "export“ the exception doesn’t occur.
- If I set the rows parameter "low enough“ so I do not get any results
  the exception doesn’t occur either.


BTW: Do you know of any tool for formatting and/or syntax highlighting
these expressions?


Christian Spitzlay





> Am 13.06.2018 um 23:02 schrieb Joel Bernstein <[hidden email]>:
>
> Can your provide some example expressions that are causing these exceptions?
>
> Joel Bernstein
> http://joelsolr.blogspot.com/
>
> On Wed, Jun 13, 2018 at 9:02 AM, Christian Spitzlay <
> [hidden email]> wrote:
>
>> Hi,
>>
>> I am seeing a lot of (reproducible) exceptions in my solr log file
>> when I execute streaming expressions:
>>
>> o.a.s.s.HttpSolrCall  Unable to write response, client closed connection
>> or we are shutting down
>> org.eclipse.jetty.io.EofException
>>        at org.eclipse.jetty.io.ChannelEndPoint.flush(
>> ChannelEndPoint.java:292)
>>        at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:429)
>>        at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:322)
>>        at org.eclipse.jetty.io.AbstractEndPoint.write(
>> AbstractEndPoint.java:372)
>>        at org.eclipse.jetty.server.HttpConnection$SendCallback.
>> process(HttpConnection.java:794)
>> […]
>>        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(
>> EatWhatYouKill.java:131)
>>        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$
>> ReservedThread.run(ReservedThreadExecutor.java:382)
>>        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
>> QueuedThreadPool.java:708)
>>        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(
>> QueuedThreadPool.java:626)
>>        at java.base/java.lang.Thread.run(Thread.java:844)
>> Caused by: java.io.IOException: Broken pipe
>>        at java.base/sun.nio.ch.FileDispatcherImpl.writev0(Native Method)
>>        at java.base/sun.nio.ch.SocketDispatcher.writev(
>> SocketDispatcher.java:51)
>>        at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:148)
>>        at java.base/sun.nio.ch.SocketChannelImpl.write(
>> SocketChannelImpl.java:506)
>>        at org.eclipse.jetty.io.ChannelEndPoint.flush(
>> ChannelEndPoint.java:272)
>>        ... 69 more
>>
>>
>> I have read up on the exception message and found
>> http://lucene.472066.n3.nabble.com/Unable-to-write-response-client-closed-
>> connection-or-we-are-shutting-down-tt4350349.html#a4350947
>> but I don’t understand how an early client connect can cause what I am
>> seeing:
>>
>> What puzzles me is that the response has been delivered in full to the
>> client library, including the document with EOF.
>>
>> So Solr must have already processed the streaming expression and returned
>> the result.
>> It’s just that the log is filled with stacktraces of this exception that
>> suggests something went wrong.
>> I don’t understand why this happens when the query seems to have succeeded.
>>
>>
>> Best regards,
>> Christian
>>
>>
>>

Reply | Threaded
Open this post in threaded view
|

Re: Exception when processing streaming expression

Susmit Shukla
Hi,
This may be expected if one of the streams is closed early - does not reach to EOF tuple

Sent from my iPhone

> On Jun 14, 2018, at 9:53 AM, Christian Spitzlay <[hidden email]> wrote:
>
> Here ist one I stripped down as far as I could:
>
> innerJoin(sort(search(kmm, q="sds_endpoint_uuid:(2f927a0b\-fe38\-451e\-9103\-580914a77e82)", fl="sds_endpoint_uuid,sds_to_endpoint_uuid", sort="sds_to_endpoint_uuid ASC", qt="/export"), by="sds_endpoint_uuid ASC"), search(kmm, q=ss_search_api_datasource:entity\:as_metadata, fl="sds_metadata_of_uuid", sort="sds_metadata_of_uuid ASC", qt="/select", rows=10000), on="sds_endpoint_uuid=sds_metadata_of_uuid")
>
> The exception happens both via PHP (search_api_solr / Solarium) and via the Solr admin UI.
> (version: Solr 7.3.1 on macOS High Sierra 10.13.5)
>
> It seems to be related to the fact that the second stream uses "select“.
> - If I use "export“ the exception doesn’t occur.
> - If I set the rows parameter "low enough“ so I do not get any results
>  the exception doesn’t occur either.
>
>
> BTW: Do you know of any tool for formatting and/or syntax highlighting
> these expressions?
>
>
> Christian Spitzlay
>
>
>
>
>
>> Am 13.06.2018 um 23:02 schrieb Joel Bernstein <[hidden email]>:
>>
>> Can your provide some example expressions that are causing these exceptions?
>>
>> Joel Bernstein
>> http://joelsolr.blogspot.com/
>>
>> On Wed, Jun 13, 2018 at 9:02 AM, Christian Spitzlay <
>> [hidden email]> wrote:
>>
>>> Hi,
>>>
>>> I am seeing a lot of (reproducible) exceptions in my solr log file
>>> when I execute streaming expressions:
>>>
>>> o.a.s.s.HttpSolrCall  Unable to write response, client closed connection
>>> or we are shutting down
>>> org.eclipse.jetty.io.EofException
>>>       at org.eclipse.jetty.io.ChannelEndPoint.flush(
>>> ChannelEndPoint.java:292)
>>>       at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:429)
>>>       at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:322)
>>>       at org.eclipse.jetty.io.AbstractEndPoint.write(
>>> AbstractEndPoint.java:372)
>>>       at org.eclipse.jetty.server.HttpConnection$SendCallback.
>>> process(HttpConnection.java:794)
>>> […]
>>>       at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(
>>> EatWhatYouKill.java:131)
>>>       at org.eclipse.jetty.util.thread.ReservedThreadExecutor$
>>> ReservedThread.run(ReservedThreadExecutor.java:382)
>>>       at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
>>> QueuedThreadPool.java:708)
>>>       at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(
>>> QueuedThreadPool.java:626)
>>>       at java.base/java.lang.Thread.run(Thread.java:844)
>>> Caused by: java.io.IOException: Broken pipe
>>>       at java.base/sun.nio.ch.FileDispatcherImpl.writev0(Native Method)
>>>       at java.base/sun.nio.ch.SocketDispatcher.writev(
>>> SocketDispatcher.java:51)
>>>       at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:148)
>>>       at java.base/sun.nio.ch.SocketChannelImpl.write(
>>> SocketChannelImpl.java:506)
>>>       at org.eclipse.jetty.io.ChannelEndPoint.flush(
>>> ChannelEndPoint.java:272)
>>>       ... 69 more
>>>
>>>
>>> I have read up on the exception message and found
>>> http://lucene.472066.n3.nabble.com/Unable-to-write-response-client-closed-
>>> connection-or-we-are-shutting-down-tt4350349.html#a4350947
>>> but I don’t understand how an early client connect can cause what I am
>>> seeing:
>>>
>>> What puzzles me is that the response has been delivered in full to the
>>> client library, including the document with EOF.
>>>
>>> So Solr must have already processed the streaming expression and returned
>>> the result.
>>> It’s just that the log is filled with stacktraces of this exception that
>>> suggests something went wrong.
>>> I don’t understand why this happens when the query seems to have succeeded.
>>>
>>>
>>> Best regards,
>>> Christian
>>>
>>>
>>>
>
Reply | Threaded
Open this post in threaded view
|

Re: Exception when processing streaming expression

Christian Spitzlay
What does that mean exactly?  If I set the rows parameter to 1000000000
the exception still occurs.  AFAICT all this happens internally during the
processing of the streaming expression.  Why wouldn't the select send
the EOF tuple when it reaches the end of the documents?
Or why wouldn't the receiving end wait for it to appear?  
Due to an incredibly low timeout used internally?


Christian Spitzlay



> Am 14.06.2018 um 19:18 schrieb Susmit <[hidden email]>:
>
> Hi,
> This may be expected if one of the streams is closed early - does not reach to EOF tuple
>
> Sent from my iPhone
>
>> On Jun 14, 2018, at 9:53 AM, Christian Spitzlay <[hidden email]> wrote:
>>
>> Here ist one I stripped down as far as I could:
>>
>> innerJoin(sort(search(kmm, q="sds_endpoint_uuid:(2f927a0b\-fe38\-451e\-9103\-580914a77e82)", fl="sds_endpoint_uuid,sds_to_endpoint_uuid", sort="sds_to_endpoint_uuid ASC", qt="/export"), by="sds_endpoint_uuid ASC"), search(kmm, q=ss_search_api_datasource:entity\:as_metadata, fl="sds_metadata_of_uuid", sort="sds_metadata_of_uuid ASC", qt="/select", rows=10000), on="sds_endpoint_uuid=sds_metadata_of_uuid")
>>
>> The exception happens both via PHP (search_api_solr / Solarium) and via the Solr admin UI.
>> (version: Solr 7.3.1 on macOS High Sierra 10.13.5)
>>
>> It seems to be related to the fact that the second stream uses "select“.
>> - If I use "export“ the exception doesn’t occur.
>> - If I set the rows parameter "low enough“ so I do not get any results
>> the exception doesn’t occur either.
>>
>>
>> BTW: Do you know of any tool for formatting and/or syntax highlighting
>> these expressions?
>>
>>
>> Christian Spitzlay
>>
>>
>>
>>
>>
>>> Am 13.06.2018 um 23:02 schrieb Joel Bernstein <[hidden email]>:
>>>
>>> Can your provide some example expressions that are causing these exceptions?
>>>
>>> Joel Bernstein
>>> http://joelsolr.blogspot.com/
>>>
>>> On Wed, Jun 13, 2018 at 9:02 AM, Christian Spitzlay <
>>> [hidden email]> wrote:
>>>
>>>> Hi,
>>>>
>>>> I am seeing a lot of (reproducible) exceptions in my solr log file
>>>> when I execute streaming expressions:
>>>>
>>>> o.a.s.s.HttpSolrCall  Unable to write response, client closed connection
>>>> or we are shutting down
>>>> org.eclipse.jetty.io.EofException
>>>>      at org.eclipse.jetty.io.ChannelEndPoint.flush(
>>>> ChannelEndPoint.java:292)
>>>>      at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:429)
>>>>      at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:322)
>>>>      at org.eclipse.jetty.io.AbstractEndPoint.write(
>>>> AbstractEndPoint.java:372)
>>>>      at org.eclipse.jetty.server.HttpConnection$SendCallback.
>>>> process(HttpConnection.java:794)
>>>> […]
>>>>      at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(
>>>> EatWhatYouKill.java:131)
>>>>      at org.eclipse.jetty.util.thread.ReservedThreadExecutor$
>>>> ReservedThread.run(ReservedThreadExecutor.java:382)
>>>>      at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
>>>> QueuedThreadPool.java:708)
>>>>      at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(
>>>> QueuedThreadPool.java:626)
>>>>      at java.base/java.lang.Thread.run(Thread.java:844)
>>>> Caused by: java.io.IOException: Broken pipe
>>>>      at java.base/sun.nio.ch.FileDispatcherImpl.writev0(Native Method)
>>>>      at java.base/sun.nio.ch.SocketDispatcher.writev(
>>>> SocketDispatcher.java:51)
>>>>      at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:148)
>>>>      at java.base/sun.nio.ch.SocketChannelImpl.write(
>>>> SocketChannelImpl.java:506)
>>>>      at org.eclipse.jetty.io.ChannelEndPoint.flush(
>>>> ChannelEndPoint.java:272)
>>>>      ... 69 more
>>>>
>>>>
>>>> I have read up on the exception message and found
>>>> http://lucene.472066.n3.nabble.com/Unable-to-write-response-client-closed-
>>>> connection-or-we-are-shutting-down-tt4350349.html#a4350947
>>>> but I don’t understand how an early client connect can cause what I am
>>>> seeing:
>>>>
>>>> What puzzles me is that the response has been delivered in full to the
>>>> client library, including the document with EOF.
>>>>
>>>> So Solr must have already processed the streaming expression and returned
>>>> the result.
>>>> It’s just that the log is filled with stacktraces of this exception that
>>>> suggests something went wrong.
>>>> I don’t understand why this happens when the query seems to have succeeded.
>>>>
>>>>
>>>> Best regards,
>>>> Christian
>>>>
>>>>
>>>>
>>

Reply | Threaded
Open this post in threaded view
|

Re: Exception when processing streaming expression

Joel Bernstein
We have to check the behavior of the innerJoin. I suspect that its closing
the second stream when the first stream his finished. This would cause a
broken pipe with the second stream. The export handler has specific code
that eats the broken pipe exception so it doesn't end up in the logs. The
select hander does not have this code.

In general you never want to use the select handler and set the rows to
such a big number. If you have that many rows you'll want to use the export
and handler which is designed to export the entire result set.

Joel Bernstein
http://joelsolr.blogspot.com/

On Thu, Jun 14, 2018 at 1:30 PM, Christian Spitzlay <
[hidden email]> wrote:

> What does that mean exactly?  If I set the rows parameter to 1000000000
> the exception still occurs.  AFAICT all this happens internally during the
> processing of the streaming expression.  Why wouldn't the select send
> the EOF tuple when it reaches the end of the documents?
> Or why wouldn't the receiving end wait for it to appear?
> Due to an incredibly low timeout used internally?
>
>
> Christian Spitzlay
>
>
>
> > Am 14.06.2018 um 19:18 schrieb Susmit <[hidden email]>:
> >
> > Hi,
> > This may be expected if one of the streams is closed early - does not
> reach to EOF tuple
> >
> > Sent from my iPhone
> >
> >> On Jun 14, 2018, at 9:53 AM, Christian Spitzlay <
> [hidden email]> wrote:
> >>
> >> Here ist one I stripped down as far as I could:
> >>
> >> innerJoin(sort(search(kmm, q="sds_endpoint_uuid:(
> 2f927a0b\-fe38\-451e\-9103\-580914a77e82)", fl="sds_endpoint_uuid,sds_to_endpoint_uuid",
> sort="sds_to_endpoint_uuid ASC", qt="/export"), by="sds_endpoint_uuid
> ASC"), search(kmm, q=ss_search_api_datasource:entity\:as_metadata,
> fl="sds_metadata_of_uuid", sort="sds_metadata_of_uuid ASC", qt="/select",
> rows=10000), on="sds_endpoint_uuid=sds_metadata_of_uuid")
> >>
> >> The exception happens both via PHP (search_api_solr / Solarium) and via
> the Solr admin UI.
> >> (version: Solr 7.3.1 on macOS High Sierra 10.13.5)
> >>
> >> It seems to be related to the fact that the second stream uses
> "select“.
> >> - If I use "export“ the exception doesn’t occur.
> >> - If I set the rows parameter "low enough“ so I do not get any results
> >> the exception doesn’t occur either.
> >>
> >>
> >> BTW: Do you know of any tool for formatting and/or syntax highlighting
> >> these expressions?
> >>
> >>
> >> Christian Spitzlay
> >>
> >>
> >>
> >>
> >>
> >>> Am 13.06.2018 um 23:02 schrieb Joel Bernstein <[hidden email]>:
> >>>
> >>> Can your provide some example expressions that are causing these
> exceptions?
> >>>
> >>> Joel Bernstein
> >>> http://joelsolr.blogspot.com/
> >>>
> >>> On Wed, Jun 13, 2018 at 9:02 AM, Christian Spitzlay <
> >>> [hidden email]> wrote:
> >>>
> >>>> Hi,
> >>>>
> >>>> I am seeing a lot of (reproducible) exceptions in my solr log file
> >>>> when I execute streaming expressions:
> >>>>
> >>>> o.a.s.s.HttpSolrCall  Unable to write response, client closed
> connection
> >>>> or we are shutting down
> >>>> org.eclipse.jetty.io.EofException
> >>>>      at org.eclipse.jetty.io.ChannelEndPoint.flush(
> >>>> ChannelEndPoint.java:292)
> >>>>      at org.eclipse.jetty.io.WriteFlusher.flush(
> WriteFlusher.java:429)
> >>>>      at org.eclipse.jetty.io.WriteFlusher.write(
> WriteFlusher.java:322)
> >>>>      at org.eclipse.jetty.io.AbstractEndPoint.write(
> >>>> AbstractEndPoint.java:372)
> >>>>      at org.eclipse.jetty.server.HttpConnection$SendCallback.
> >>>> process(HttpConnection.java:794)
> >>>> […]
> >>>>      at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(
> >>>> EatWhatYouKill.java:131)
> >>>>      at org.eclipse.jetty.util.thread.ReservedThreadExecutor$
> >>>> ReservedThread.run(ReservedThreadExecutor.java:382)
> >>>>      at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
> >>>> QueuedThreadPool.java:708)
> >>>>      at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(
> >>>> QueuedThreadPool.java:626)
> >>>>      at java.base/java.lang.Thread.run(Thread.java:844)
> >>>> Caused by: java.io.IOException: Broken pipe
> >>>>      at java.base/sun.nio.ch.FileDispatcherImpl.writev0(Native
> Method)
> >>>>      at java.base/sun.nio.ch.SocketDispatcher.writev(
> >>>> SocketDispatcher.java:51)
> >>>>      at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:148)
> >>>>      at java.base/sun.nio.ch.SocketChannelImpl.write(
> >>>> SocketChannelImpl.java:506)
> >>>>      at org.eclipse.jetty.io.ChannelEndPoint.flush(
> >>>> ChannelEndPoint.java:272)
> >>>>      ... 69 more
> >>>>
> >>>>
> >>>> I have read up on the exception message and found
> >>>> http://lucene.472066.n3.nabble.com/Unable-to-write-
> response-client-closed-
> >>>> connection-or-we-are-shutting-down-tt4350349.html#a4350947
> >>>> but I don’t understand how an early client connect can cause what I am
> >>>> seeing:
> >>>>
> >>>> What puzzles me is that the response has been delivered in full to the
> >>>> client library, including the document with EOF.
> >>>>
> >>>> So Solr must have already processed the streaming expression and
> returned
> >>>> the result.
> >>>> It’s just that the log is filled with stacktraces of this exception
> that
> >>>> suggests something went wrong.
> >>>> I don’t understand why this happens when the query seems to have
> succeeded.
> >>>>
> >>>>
> >>>> Best regards,
> >>>> Christian
> >>>>
> >>>>
> >>>>
> >>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Exception when processing streaming expression

Christian Spitzlay

> Am 15.06.2018 um 01:23 schrieb Joel Bernstein <[hidden email]>:
>
> We have to check the behavior of the innerJoin. I suspect that its closing
> the second stream when the first stream his finished. This would cause a
> broken pipe with the second stream. The export handler has specific code
> that eats the broken pipe exception so it doesn't end up in the logs. The
> select hander does not have this code.

Ah, I see.  The stack trace in my original mail has the "broken pipe" message:

[...]
Caused by: java.io.IOException: Broken pipe
        at java.base/sun.nio.ch.FileDispatcherImpl.writev0(Native Method)
[...]


> In general you never want to use the select handler and set the rows to
> such a big number. If you have that many rows you'll want to use the export
> and handler which is designed to export the entire result set.


We started out with the export handler but we are updating documents using
streaming expressions and we had fields that had types
that do not support docValues, according to the documentation at
https://lucene.apache.org/solr/guide/7_3/docvalues.html#enabling-docvalues

We switched to the select handler in some places and it worked.
We set the rows parameter to a large value:
"If you want to tell Solr to return all possible results from the query without an
upper bound, specify rows to be 10000000 or some other ridiculously
large value that is higher than the possible number of rows that are expected."
From:
https://wiki.apache.org/solr/CommonQueryParameters#rows


Christian Spitzlay