[jira] Created: (SOLR-1711) Race condition in org/apache/solr/client/solrj/impl/StreamingUpdateSolrServer.java

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

[jira] Created: (SOLR-1711) Race condition in org/apache/solr/client/solrj/impl/StreamingUpdateSolrServer.java

JIRA jira@apache.org
Race condition in org/apache/solr/client/solrj/impl/StreamingUpdateSolrServer.java
----------------------------------------------------------------------------------

                 Key: SOLR-1711
                 URL: https://issues.apache.org/jira/browse/SOLR-1711
             Project: Solr
          Issue Type: Bug
          Components: clients - java
    Affects Versions: 1.4, 1.5
            Reporter: Attila Babo
            Priority: Critical
             Fix For: 1.5


While inserting a large pile of documents using StreamingUpdateSolrServer there is a race condition as all Runner instances stop processing while the blocking queue is full. With a high performance client this could happen quite often, there is no way to recover from it at the client side.

In StreamingUpdateSolrServer there is a BlockingQueue called queue to store UpdateRequests, there are up to threadCount number of workers threads from StreamingUpdateSolrServer.Runner to read that queue and push requests to a Solr instance. If at one point the BlockingQueue is empty all workers stop processing it and pushing the collected content to Solr which could be a time consuming process, sometimes all worker threads are waiting for Solr. If at this time the client fills the BlockingQueue to full all worker threads will quit without processing any further and the main thread will block forever.

There is a simple, well tested patch handle this situation.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (SOLR-1711) Race condition in org/apache/solr/client/solrj/impl/StreamingUpdateSolrServer.java

JIRA jira@apache.org

     [ https://issues.apache.org/jira/browse/SOLR-1711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Attila Babo updated SOLR-1711:
------------------------------

    Attachment: StreamingUpdateSolrServer.patch

Patch 1, 2:
Inside the Runner.run method I've added a do while loop to prevent the Runner to quit while there are new requests, this handles the problem of new requests added while Runner is sending the previous batch.

Patch 3
Validity check of method variable is not strictly necessary, just a code clean up.

Patch 4
The last part of the patch is to move synchronized outside of conditional to avoid a situation where runners change while evaluating it.

To minify the patch all indentation has been removed.

> Race condition in org/apache/solr/client/solrj/impl/StreamingUpdateSolrServer.java
> ----------------------------------------------------------------------------------
>
>                 Key: SOLR-1711
>                 URL: https://issues.apache.org/jira/browse/SOLR-1711
>             Project: Solr
>          Issue Type: Bug
>          Components: clients - java
>    Affects Versions: 1.4, 1.5
>            Reporter: Attila Babo
>            Priority: Critical
>             Fix For: 1.5
>
>         Attachments: StreamingUpdateSolrServer.patch
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> While inserting a large pile of documents using StreamingUpdateSolrServer there is a race condition as all Runner instances stop processing while the blocking queue is full. With a high performance client this could happen quite often, there is no way to recover from it at the client side.
> In StreamingUpdateSolrServer there is a BlockingQueue called queue to store UpdateRequests, there are up to threadCount number of workers threads from StreamingUpdateSolrServer.Runner to read that queue and push requests to a Solr instance. If at one point the BlockingQueue is empty all workers stop processing it and pushing the collected content to Solr which could be a time consuming process, sometimes all worker threads are waiting for Solr. If at this time the client fills the BlockingQueue to full all worker threads will quit without processing any further and the main thread will block forever.
> There is a simple, well tested patch handle this situation.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (SOLR-1711) Race condition in org/apache/solr/client/solrj/impl/StreamingUpdateSolrServer.java

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

     [ https://issues.apache.org/jira/browse/SOLR-1711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Attila Babo updated SOLR-1711:
------------------------------

    Description:
While inserting a large pile of documents using StreamingUpdateSolrServer there is a race condition as all Runner instances stop processing while the blocking queue is full. With a high performance client this could happen quite often, there is no way to recover from it at the client side.

In StreamingUpdateSolrServer there is a BlockingQueue called queue to store UpdateRequests, there are up to threadCount number of workers threads from StreamingUpdateSolrServer.Runner to read that queue and push requests to a Solr instance. If at one point the BlockingQueue is empty all workers stop processing it and pushing the collected content to Solr which could be a time consuming process, sometimes all worker threads are waiting for Solr. If at this time the client fills the BlockingQueue to full all worker threads will quit without processing any further and the main thread will block forever.

There is a simple, well tested patch attached to handle this situation.

  was:
While inserting a large pile of documents using StreamingUpdateSolrServer there is a race condition as all Runner instances stop processing while the blocking queue is full. With a high performance client this could happen quite often, there is no way to recover from it at the client side.

In StreamingUpdateSolrServer there is a BlockingQueue called queue to store UpdateRequests, there are up to threadCount number of workers threads from StreamingUpdateSolrServer.Runner to read that queue and push requests to a Solr instance. If at one point the BlockingQueue is empty all workers stop processing it and pushing the collected content to Solr which could be a time consuming process, sometimes all worker threads are waiting for Solr. If at this time the client fills the BlockingQueue to full all worker threads will quit without processing any further and the main thread will block forever.

There is a simple, well tested patch handle this situation.


> Race condition in org/apache/solr/client/solrj/impl/StreamingUpdateSolrServer.java
> ----------------------------------------------------------------------------------
>
>                 Key: SOLR-1711
>                 URL: https://issues.apache.org/jira/browse/SOLR-1711
>             Project: Solr
>          Issue Type: Bug
>          Components: clients - java
>    Affects Versions: 1.4, 1.5
>            Reporter: Attila Babo
>            Priority: Critical
>             Fix For: 1.5
>
>         Attachments: StreamingUpdateSolrServer.patch
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> While inserting a large pile of documents using StreamingUpdateSolrServer there is a race condition as all Runner instances stop processing while the blocking queue is full. With a high performance client this could happen quite often, there is no way to recover from it at the client side.
> In StreamingUpdateSolrServer there is a BlockingQueue called queue to store UpdateRequests, there are up to threadCount number of workers threads from StreamingUpdateSolrServer.Runner to read that queue and push requests to a Solr instance. If at one point the BlockingQueue is empty all workers stop processing it and pushing the collected content to Solr which could be a time consuming process, sometimes all worker threads are waiting for Solr. If at this time the client fills the BlockingQueue to full all worker threads will quit without processing any further and the main thread will block forever.
> There is a simple, well tested patch attached to handle this situation.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

[jira] Resolved: (SOLR-1711) Race condition in org/apache/solr/client/solrj/impl/StreamingUpdateSolrServer.java

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

     [ https://issues.apache.org/jira/browse/SOLR-1711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Yonik Seeley resolved SOLR-1711.
--------------------------------

    Resolution: Fixed

Thanks Attila!  I just committed this.

> Race condition in org/apache/solr/client/solrj/impl/StreamingUpdateSolrServer.java
> ----------------------------------------------------------------------------------
>
>                 Key: SOLR-1711
>                 URL: https://issues.apache.org/jira/browse/SOLR-1711
>             Project: Solr
>          Issue Type: Bug
>          Components: clients - java
>    Affects Versions: 1.4, 1.5
>            Reporter: Attila Babo
>            Priority: Critical
>             Fix For: 1.5
>
>         Attachments: StreamingUpdateSolrServer.patch
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> While inserting a large pile of documents using StreamingUpdateSolrServer there is a race condition as all Runner instances stop processing while the blocking queue is full. With a high performance client this could happen quite often, there is no way to recover from it at the client side.
> In StreamingUpdateSolrServer there is a BlockingQueue called queue to store UpdateRequests, there are up to threadCount number of workers threads from StreamingUpdateSolrServer.Runner to read that queue and push requests to a Solr instance. If at one point the BlockingQueue is empty all workers stop processing it and pushing the collected content to Solr which could be a time consuming process, sometimes all worker threads are waiting for Solr. If at this time the client fills the BlockingQueue to full all worker threads will quit without processing any further and the main thread will block forever.
> There is a simple, well tested patch attached to handle this situation.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (SOLR-1711) Race condition in org/apache/solr/client/solrj/impl/StreamingUpdateSolrServer.java

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/SOLR-1711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12849865#action_12849865 ]

Luke Forehand commented on SOLR-1711:
-------------------------------------

This is a very serious problem for us.  We have multiple threads adding to the StreamingUpdateSolrServer's BlockingQueue, and if I bump the thread count high enough (around 10 for my process) I can reproduce this problem every time.  I'd say this bug is critical enough to warrant a SOLR bug-fix release.

> Race condition in org/apache/solr/client/solrj/impl/StreamingUpdateSolrServer.java
> ----------------------------------------------------------------------------------
>
>                 Key: SOLR-1711
>                 URL: https://issues.apache.org/jira/browse/SOLR-1711
>             Project: Solr
>          Issue Type: Bug
>          Components: clients - java
>    Affects Versions: 1.4, 1.5
>            Reporter: Attila Babo
>            Priority: Critical
>             Fix For: 1.5
>
>         Attachments: StreamingUpdateSolrServer.patch
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> While inserting a large pile of documents using StreamingUpdateSolrServer there is a race condition as all Runner instances stop processing while the blocking queue is full. With a high performance client this could happen quite often, there is no way to recover from it at the client side.
> In StreamingUpdateSolrServer there is a BlockingQueue called queue to store UpdateRequests, there are up to threadCount number of workers threads from StreamingUpdateSolrServer.Runner to read that queue and push requests to a Solr instance. If at one point the BlockingQueue is empty all workers stop processing it and pushing the collected content to Solr which could be a time consuming process, sometimes all worker threads are waiting for Solr. If at this time the client fills the BlockingQueue to full all worker threads will quit without processing any further and the main thread will block forever.
> There is a simple, well tested patch attached to handle this situation.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.