2 Async exceptions during distributed update issue...

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

2 Async exceptions during distributed update issue...

Fiz Ahmed


Hello  Solr Experts, 


Just wanted to follow up in case my question.Would appreciate help on this.

 

SOLR Version : 6.6.2

OS – Linux 3.1.2

JDK – 1.8

 

Shard – 16 – All are active.

Xms – 16 gb

Xmx – 16 gb

Host has 64 cores.

 

the update processor chain.attaching the complete updateRequestProcessorChain in a file.

Attaching physical and CPU memory screenshot.

 

There are multiple threads sending products to solr. tested with batch size per thread 50 & 100, it worked fine without error and if the batch size is 1000 the following error occurs.

 

 

I am getting the following error when the batch size is 1000. Please advise.

 

2019-11-14T17:45:44,646 - ERROR [qtp876213901-1169:SolrException@YYY] - {collection=c:ducts, core=x:ducts_shard15_replica1, node_name=n:10.YYY.40.81:8983_solr, replica=r:core_node30, shard=s:shard15} - org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException: 2 Async exceptions during distributed update:

10.YYY.40.62:8983 failed to respond

10.YYY.40.62:8983 failed to respond

 

2019-11-14T17:45:48,821 - ERROR [qtp876213901-202:SolrException@YYY] - {collection=c:ducts, core=x:ducts_shard7_replica1, node_name=n:10.YYY.40.81:8983_solr, replica=r:core_node29, shard=s:shard7} - org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException: Async exception during distributed update: 10.YYY.40.81:8983 failed to respond

 

 

2019-11-14T19:36:11,599 - ERROR [updateExecutor-2-thread-176-processing-x:ducts_shard3_replica2 http:////10.YYY.40.68:8983//solr//ducts_shard11_replica2 r:core_node26 n:10.YYY.40.68:8983_solr c:ducts s:shard3:StreamingSolrClients$ErrorReportingConcurrentUpdateSolrClient@131] - {ConcurrentUpdateSolrClient.url=http://10.YYY.40.68:8983/solr/ducts_shard11_replica2, collection=c:ducts, core=x:ducts_shard3_replica2, node_name=n:10.YYY.40.68:8983_solr, replica=r:core_node26, shard=s:shard3} - error

org.apache.http.NoHttpResponseException: 10.YYY.40.68:8983 failed to respond

        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143) ~[httpclient-4.4.1.jar:4.4.1]

        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57) ~[httpclient-4.4.1.jar:4.4.1]

        at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) ~[httpcore-4.4.1.jar:4.4.1]

 

 

2019-11-14T19:36:14,567 - ERROR [updateExecutor-2-thread-189-processing-x:ducts_shard11_replica2 http:////10.YYY.40.62:8983//solr//ducts_shard2_replica1 r:core_node25 n:10.YYY.40.68:8983_solr c:ducts s:shard11:StreamingSolrClients$ErrorReportingConcurrentUpdateSolrClient@131] - {ConcurrentUpdateSolrClient.url=http://10.YYY.40.62:8983/solr/ducts_shard2_replica1, collection=c:ducts, core=x:ducts_shard11_replica2, node_name=n:10.YYY.40.68:8983_solr, replica=r:core_node25, shard=s:shard11} - error

java.net.SocketException: Broken pipe (Write failed)

        at java.net.SocketOutputStream.socketWrite0(Native Method) ~[?:1.8.0_232]

        at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111) ~[?:1.8.0_232]

        at java.net.SocketOutputStream.write(SocketOutputStream.java:155) ~[?:1.8.0_232]

 

 

2019-11-14T19:36:38,851 - ERROR [qtp876213901-542:SolrException@159] - {collection=c:ducts, core=x:ducts_shard11_replica2, node_name=n:10.YYY.40.68:8983_solr, replica=r:core_node25, shard=s:shard11} - null:org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException: Async exception during distributed update: Broken pipe (Write failed)

       at org.apache.solr.update.processor.DistributedUpdateProcessor.doFinish(DistributedUpdateProcessor.java:972)

        at org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1911)

 

 

Thanks 

 


sample.txt (7K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: 2 Async exceptions during distributed update issue...

Erick Erickson
First, please don’t use the “schemaless” mode (add-unknown-fields-to-schema in your solrconfig) while load testing. There’s quite a bit of work being done when Solr discovers an unknown field that’ll cause some instability under heavy load.

Second, when you put a large batch against Solr, you have the possibility of the update just taking longer than the timeout. There are several timeouts you can increase, see the “solr.xml” section of the ref guide.

Best,
Erick

> On Nov 19, 2019, at 12:29 PM, Fiz N <[hidden email]> wrote:
>
>
> Hello  Solr Experts,
>
> Just wanted to follow up in case my question.Would appreciate help on this.
>  
> SOLR Version : 6.6.2
> OS – Linux 3.1.2
> JDK – 1.8
>  
> Shard – 16 – All are active.
> Xms – 16 gb
> Xmx – 16 gb
> Host has 64 cores.
>  
> the update processor chain.attaching the complete updateRequestProcessorChain in a file.
> Attaching physical and CPU memory screenshot.
>  
> There are multiple threads sending products to solr. tested with batch size per thread 50 & 100, it worked fine without error and if the batch size is 1000 the following error occurs.
>  
>  
> I am getting the following error when the batch size is 1000. Please advise.
>  
> 2019-11-14T17:45:44,646 - ERROR [qtp876213901-1169:SolrException@YYY] - {collection=c:ducts, core=x:ducts_shard15_replica1, node_name=n:10.YYY.40.81:8983_solr, replica=r:core_node30, shard=s:shard15} - org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException: 2 Async exceptions during distributed update:
> 10.YYY.40.62:8983 failed to respond
> 10.YYY.40.62:8983 failed to respond
>  
> 2019-11-14T17:45:48,821 - ERROR [qtp876213901-202:SolrException@YYY] - {collection=c:ducts, core=x:ducts_shard7_replica1, node_name=n:10.YYY.40.81:8983_solr, replica=r:core_node29, shard=s:shard7} - org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException: Async exception during distributed update: 10.YYY.40.81:8983 failed to respond
>  
>  
> 2019-11-14T19:36:11,599 - ERROR [updateExecutor-2-thread-176-processing-x:ducts_shard3_replica2 http:////10.YYY.40.68:8983//solr//ducts_shard11_replica2 r:core_node26 n:10.YYY.40.68:8983_solr c:ducts s:shard3:StreamingSolrClients$ErrorReportingConcurrentUpdateSolrClient@131] - {ConcurrentUpdateSolrClient.url=http://10.YYY.40.68:8983/solr/ducts_shard11_replica2, collection=c:ducts, core=x:ducts_shard3_replica2, node_name=n:10.YYY.40.68:8983_solr, replica=r:core_node26, shard=s:shard3} - error
> org.apache.http.NoHttpResponseException: 10.YYY.40.68:8983 failed to respond
>         at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143) ~[httpclient-4.4.1.jar:4.4.1]
>         at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57) ~[httpclient-4.4.1.jar:4.4.1]
>         at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) ~[httpcore-4.4.1.jar:4.4.1]
>  
>  
> 2019-11-14T19:36:14,567 - ERROR [updateExecutor-2-thread-189-processing-x:ducts_shard11_replica2 http:////10.YYY.40.62:8983//solr//ducts_shard2_replica1 r:core_node25 n:10.YYY.40.68:8983_solr c:ducts s:shard11:StreamingSolrClients$ErrorReportingConcurrentUpdateSolrClient@131] - {ConcurrentUpdateSolrClient.url=http://10.YYY.40.62:8983/solr/ducts_shard2_replica1, collection=c:ducts, core=x:ducts_shard11_replica2, node_name=n:10.YYY.40.68:8983_solr, replica=r:core_node25, shard=s:shard11} - error
> java.net.SocketException: Broken pipe (Write failed)
>         at java.net.SocketOutputStream.socketWrite0(Native Method) ~[?:1.8.0_232]
>         at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111) ~[?:1.8.0_232]
>         at java.net.SocketOutputStream.write(SocketOutputStream.java:155) ~[?:1.8.0_232]
>  
>  
> 2019-11-14T19:36:38,851 - ERROR [qtp876213901-542:SolrException@159] - {collection=c:ducts, core=x:ducts_shard11_replica2, node_name=n:10.YYY.40.68:8983_solr, replica=r:core_node25, shard=s:shard11} - null:org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException: Async exception during distributed update: Broken pipe (Write failed)
>        at org.apache.solr.update.processor.DistributedUpdateProcessor.doFinish(DistributedUpdateProcessor.java:972)
>         at org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1911)
>  
>  
> Thanks
>  
> <sample.txt>