[jira] [Created] (SOLR-3284) StreamingUpdateSolrServer swallows exceptions

classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

[jira] [Created] (SOLR-3284) StreamingUpdateSolrServer swallows exceptions

Markus Jelsma (Jira)
StreamingUpdateSolrServer swallows exceptions
---------------------------------------------

                 Key: SOLR-3284
                 URL: https://issues.apache.org/jira/browse/SOLR-3284
             Project: Solr
          Issue Type: Improvement
          Components: clients - java
    Affects Versions: 3.5, 4.0
            Reporter: Shawn Heisey


StreamingUpdateSolrServer eats exceptions thrown by lower level code, such as HttpClient, when doing adds.  It may happen with other methods, though I know that query and deleteByQuery will throw exceptions.  I believe that this is a result of the queue/Runner design.  That's what makes SUSS perform better, but it means you sacrifice the ability to programmatically determine that there was a problem with your update.  All errors are logged via slf4j, but that's not terribly helpful except with determining what went wrong after the fact.

When using CommonsHttpSolrServer, I've been able to rely on getting an exception thrown by pretty much any error, letting me use try/catch to detect problems.

There's probably enough dependent code out there that it would not be a good idea to change the design of SUSS, unless there were alternate constructors or additional methods available to configure new/old behavior.  Fixing this is probably not trivial, so it's probably a better idea to come up with a new server object based on CHSS.  This is outside my current skillset.


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] [Commented] (SOLR-3284) StreamingUpdateSolrServer swallows exceptions

Markus Jelsma (Jira)

    [ https://issues.apache.org/jira/browse/SOLR-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13255433#comment-13255433 ]

Sami Siren commented on SOLR-3284:
----------------------------------

What are the operations/error situations where you are not seeing an Exception when you expect one?

By default the ConcurrentUpdateSolrServer (StreamingUpdateSolrServer) just logs the exceptions from updates but you can override this functionality:

{code}
    SolrServer server = new ConcurrentUpdateSolrServer("http://127.0.0.1:8983/solr", 1, 1){
      public void handleError(Throwable ex) {
        //do something with the Throwable here
        System.out.println("Something wrong!" + ex.getMessage());
      }
    };
   
    server.add(new SolrInputDocument());

{code}

The current exception reporting is pretty limited and it is impossible to see which operation triggered the exception but such improvements should be done in separate issues.
               

> StreamingUpdateSolrServer swallows exceptions
> ---------------------------------------------
>
>                 Key: SOLR-3284
>                 URL: https://issues.apache.org/jira/browse/SOLR-3284
>             Project: Solr
>          Issue Type: Improvement
>          Components: clients - java
>    Affects Versions: 3.5, 4.0
>            Reporter: Shawn Heisey
>
> StreamingUpdateSolrServer eats exceptions thrown by lower level code, such as HttpClient, when doing adds.  It may happen with other methods, though I know that query and deleteByQuery will throw exceptions.  I believe that this is a result of the queue/Runner design.  That's what makes SUSS perform better, but it means you sacrifice the ability to programmatically determine that there was a problem with your update.  All errors are logged via slf4j, but that's not terribly helpful except with determining what went wrong after the fact.
> When using CommonsHttpSolrServer, I've been able to rely on getting an exception thrown by pretty much any error, letting me use try/catch to detect problems.
> There's probably enough dependent code out there that it would not be a good idea to change the design of SUSS, unless there were alternate constructors or additional methods available to configure new/old behavior.  Fixing this is probably not trivial, so it's probably a better idea to come up with a new server object based on CHSS.  This is outside my current skillset.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] [Commented] (SOLR-3284) StreamingUpdateSolrServer swallows exceptions

Markus Jelsma (Jira)
In reply to this post by Markus Jelsma (Jira)

    [ https://issues.apache.org/jira/browse/SOLR-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13255602#comment-13255602 ]

Shawn Heisey commented on SOLR-3284:
------------------------------------

If the Solr server goes down in between updates done with the concurrent server, doing further updates will fail, but the calling code will not know that.  With the Commons or Http server, an exception is thrown that my code catches.

I don't think that just overriding handleError is enough.  If Solr goes down but the machine is still up, you have immediate failure detection because the connection will be refused.  If the server goes away entirely, it could take a couple of minutes to fail.  You would have to provide methods to check that 1) all background operations are complete and 2) they were error free.

I can no longer remember whether an exception is thrown when trying a commit against a down machine with the concurrent server.  IIRC it does throw one in this instance.  I definitely believe that it should.  Perhaps the current handleError code could update class-level members (with names like "boolean updateErrored" and "SolrServerException updateException") that could be checked and used by the commit method.  If they are set, it would reset them and throw an exception (fast-fail) without actually trying the commit.  There should probably be a constructor option and a set method to either activate this new behavior or restore the original behavior.

When I first designed my code, I was relying on the exceptions thrown by the commons server when doing the actual update, so it's too late by the time it reaches the commit - it has already updated the position values.  I now realize that this is incorrect design, though I might never have figured it out without my attempt to use the concurrent server.  It's going to be a bit painful to redesign my code to put off updating position values until after a successful commit operation.  It's something I do intend to do.

               

> StreamingUpdateSolrServer swallows exceptions
> ---------------------------------------------
>
>                 Key: SOLR-3284
>                 URL: https://issues.apache.org/jira/browse/SOLR-3284
>             Project: Solr
>          Issue Type: Improvement
>          Components: clients - java
>    Affects Versions: 3.5, 4.0
>            Reporter: Shawn Heisey
>
> StreamingUpdateSolrServer eats exceptions thrown by lower level code, such as HttpClient, when doing adds.  It may happen with other methods, though I know that query and deleteByQuery will throw exceptions.  I believe that this is a result of the queue/Runner design.  That's what makes SUSS perform better, but it means you sacrifice the ability to programmatically determine that there was a problem with your update.  All errors are logged via slf4j, but that's not terribly helpful except with determining what went wrong after the fact.
> When using CommonsHttpSolrServer, I've been able to rely on getting an exception thrown by pretty much any error, letting me use try/catch to detect problems.
> There's probably enough dependent code out there that it would not be a good idea to change the design of SUSS, unless there were alternate constructors or additional methods available to configure new/old behavior.  Fixing this is probably not trivial, so it's probably a better idea to come up with a new server object based on CHSS.  This is outside my current skillset.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] [Updated] (SOLR-3284) StreamingUpdateSolrServer swallows exceptions

Markus Jelsma (Jira)
In reply to this post by Markus Jelsma (Jira)

     [ https://issues.apache.org/jira/browse/SOLR-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Shawn Heisey updated SOLR-3284:
-------------------------------

    Attachment: SOLR-3284.patch

First crack at a patch for throwing delayed exceptions.  It should do this on any request when a previous request resulted in an error, not just on commits.
               

> StreamingUpdateSolrServer swallows exceptions
> ---------------------------------------------
>
>                 Key: SOLR-3284
>                 URL: https://issues.apache.org/jira/browse/SOLR-3284
>             Project: Solr
>          Issue Type: Improvement
>          Components: clients - java
>    Affects Versions: 3.5, 4.0
>            Reporter: Shawn Heisey
>         Attachments: SOLR-3284.patch
>
>
> StreamingUpdateSolrServer eats exceptions thrown by lower level code, such as HttpClient, when doing adds.  It may happen with other methods, though I know that query and deleteByQuery will throw exceptions.  I believe that this is a result of the queue/Runner design.  That's what makes SUSS perform better, but it means you sacrifice the ability to programmatically determine that there was a problem with your update.  All errors are logged via slf4j, but that's not terribly helpful except with determining what went wrong after the fact.
> When using CommonsHttpSolrServer, I've been able to rely on getting an exception thrown by pretty much any error, letting me use try/catch to detect problems.
> There's probably enough dependent code out there that it would not be a good idea to change the design of SUSS, unless there were alternate constructors or additional methods available to configure new/old behavior.  Fixing this is probably not trivial, so it's probably a better idea to come up with a new server object based on CHSS.  This is outside my current skillset.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] [Commented] (SOLR-3284) StreamingUpdateSolrServer swallows exceptions

Markus Jelsma (Jira)
In reply to this post by Markus Jelsma (Jira)

    [ https://issues.apache.org/jira/browse/SOLR-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13255659#comment-13255659 ]

Shawn Heisey commented on SOLR-3284:
------------------------------------

The patch should also apply successfully to 3.6.
               

> StreamingUpdateSolrServer swallows exceptions
> ---------------------------------------------
>
>                 Key: SOLR-3284
>                 URL: https://issues.apache.org/jira/browse/SOLR-3284
>             Project: Solr
>          Issue Type: Improvement
>          Components: clients - java
>    Affects Versions: 3.5, 4.0
>            Reporter: Shawn Heisey
>         Attachments: SOLR-3284.patch
>
>
> StreamingUpdateSolrServer eats exceptions thrown by lower level code, such as HttpClient, when doing adds.  It may happen with other methods, though I know that query and deleteByQuery will throw exceptions.  I believe that this is a result of the queue/Runner design.  That's what makes SUSS perform better, but it means you sacrifice the ability to programmatically determine that there was a problem with your update.  All errors are logged via slf4j, but that's not terribly helpful except with determining what went wrong after the fact.
> When using CommonsHttpSolrServer, I've been able to rely on getting an exception thrown by pretty much any error, letting me use try/catch to detect problems.
> There's probably enough dependent code out there that it would not be a good idea to change the design of SUSS, unless there were alternate constructors or additional methods available to configure new/old behavior.  Fixing this is probably not trivial, so it's probably a better idea to come up with a new server object based on CHSS.  This is outside my current skillset.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] [Issue Comment Edited] (SOLR-3284) StreamingUpdateSolrServer swallows exceptions

Markus Jelsma (Jira)
In reply to this post by Markus Jelsma (Jira)

    [ https://issues.apache.org/jira/browse/SOLR-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13255657#comment-13255657 ]

Shawn Heisey edited comment on SOLR-3284 at 4/17/12 3:45 PM:
-------------------------------------------------------------

First crack at a patch for throwing delayed exceptions.  It should do this on any request when a previous request resulted in an error, not just on commits.  I did not attempt to make any unit tests.  I'm not entirely sure how unit tests work when things are supposed to succeed, how to simulate a failure is even less obvious.
               
      was (Author: elyograg):
    First crack at a patch for throwing delayed exceptions.  It should do this on any request when a previous request resulted in an error, not just on commits.
                 

> StreamingUpdateSolrServer swallows exceptions
> ---------------------------------------------
>
>                 Key: SOLR-3284
>                 URL: https://issues.apache.org/jira/browse/SOLR-3284
>             Project: Solr
>          Issue Type: Improvement
>          Components: clients - java
>    Affects Versions: 3.5, 4.0
>            Reporter: Shawn Heisey
>         Attachments: SOLR-3284.patch
>
>
> StreamingUpdateSolrServer eats exceptions thrown by lower level code, such as HttpClient, when doing adds.  It may happen with other methods, though I know that query and deleteByQuery will throw exceptions.  I believe that this is a result of the queue/Runner design.  That's what makes SUSS perform better, but it means you sacrifice the ability to programmatically determine that there was a problem with your update.  All errors are logged via slf4j, but that's not terribly helpful except with determining what went wrong after the fact.
> When using CommonsHttpSolrServer, I've been able to rely on getting an exception thrown by pretty much any error, letting me use try/catch to detect problems.
> There's probably enough dependent code out there that it would not be a good idea to change the design of SUSS, unless there were alternate constructors or additional methods available to configure new/old behavior.  Fixing this is probably not trivial, so it's probably a better idea to come up with a new server object based on CHSS.  This is outside my current skillset.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] [Commented] (SOLR-3284) StreamingUpdateSolrServer swallows exceptions

Markus Jelsma (Jira)
In reply to this post by Markus Jelsma (Jira)

    [ https://issues.apache.org/jira/browse/SOLR-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13255705#comment-13255705 ]

Shawn Heisey commented on SOLR-3284:
------------------------------------

After looking at existing tests to see how I might implement tests for this new functionality, I couldn't see how to do it.  Also, I noticed that there are tests for SolrCloud and something else called ChaosMonkey.  All tests in solr/ pass with this patch, but I don't know how SolrCloud might be affected.  I would hope that it already handles exceptions properly and therefore wouldn't have any problems, but I have never looked at the code or used SolrCloud.

               

> StreamingUpdateSolrServer swallows exceptions
> ---------------------------------------------
>
>                 Key: SOLR-3284
>                 URL: https://issues.apache.org/jira/browse/SOLR-3284
>             Project: Solr
>          Issue Type: Improvement
>          Components: clients - java
>    Affects Versions: 3.5, 4.0
>            Reporter: Shawn Heisey
>         Attachments: SOLR-3284.patch
>
>
> StreamingUpdateSolrServer eats exceptions thrown by lower level code, such as HttpClient, when doing adds.  It may happen with other methods, though I know that query and deleteByQuery will throw exceptions.  I believe that this is a result of the queue/Runner design.  That's what makes SUSS perform better, but it means you sacrifice the ability to programmatically determine that there was a problem with your update.  All errors are logged via slf4j, but that's not terribly helpful except with determining what went wrong after the fact.
> When using CommonsHttpSolrServer, I've been able to rely on getting an exception thrown by pretty much any error, letting me use try/catch to detect problems.
> There's probably enough dependent code out there that it would not be a good idea to change the design of SUSS, unless there were alternate constructors or additional methods available to configure new/old behavior.  Fixing this is probably not trivial, so it's probably a better idea to come up with a new server object based on CHSS.  This is outside my current skillset.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] [Commented] (SOLR-3284) StreamingUpdateSolrServer swallows exceptions

Markus Jelsma (Jira)
In reply to this post by Markus Jelsma (Jira)

    [ https://issues.apache.org/jira/browse/SOLR-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13292206#comment-13292206 ]

Lance Norskog commented on SOLR-3284:
-------------------------------------

In the trunk it is now ConcurrentUpdateSolrServer, and it swallows errors all over the place.

I am a total fascist about fail-fast/fail-loud. A more obnoxious (fail-fast) design is:
* Any exception in a worker causes the main driver to kill all other workers.
* The default handler after this is a rollback.
* Add a flush() method to SolrServer.
** CUSS uses this to report back to the main thread that everything blew up.
** Other server classes would do a socket flush and wait and drain out any error text.
* Use a finalize() method to complain that the user did not call flush().
** This is in all SolrServer classes- the same application code should do the same thing with every SolrServer implementation.

We tell our customers to not use SUSS/CUSS because it does not do errors right. Partly this causes problems because people tend to "design to success" (assume everything works) instead of "design to failure" (assume everything breaks). When CUSS breaks, the production scripting and staff do not notice.


               

> StreamingUpdateSolrServer swallows exceptions
> ---------------------------------------------
>
>                 Key: SOLR-3284
>                 URL: https://issues.apache.org/jira/browse/SOLR-3284
>             Project: Solr
>          Issue Type: Improvement
>          Components: clients - java
>    Affects Versions: 3.5, 4.0
>            Reporter: Shawn Heisey
>         Attachments: SOLR-3284.patch
>
>
> StreamingUpdateSolrServer eats exceptions thrown by lower level code, such as HttpClient, when doing adds.  It may happen with other methods, though I know that query and deleteByQuery will throw exceptions.  I believe that this is a result of the queue/Runner design.  That's what makes SUSS perform better, but it means you sacrifice the ability to programmatically determine that there was a problem with your update.  All errors are logged via slf4j, but that's not terribly helpful except with determining what went wrong after the fact.
> When using CommonsHttpSolrServer, I've been able to rely on getting an exception thrown by pretty much any error, letting me use try/catch to detect problems.
> There's probably enough dependent code out there that it would not be a good idea to change the design of SUSS, unless there were alternate constructors or additional methods available to configure new/old behavior.  Fixing this is probably not trivial, so it's probably a better idea to come up with a new server object based on CHSS.  This is outside my current skillset.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] [Commented] (SOLR-3284) StreamingUpdateSolrServer swallows exceptions

Markus Jelsma (Jira)
In reply to this post by Markus Jelsma (Jira)

    [ https://issues.apache.org/jira/browse/SOLR-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13292507#comment-13292507 ]

Mike Sokolov commented on SOLR-3284:
------------------------------------

Another approach we have been using in a similar parallel writing situation is to have the thread pool maintain a fixed-size list of exceptions.  Whenever a worker throws one, it gets put on the list.  Then at certain barrier conditions (flush, exception list full), the exceptions are collected together and re-thrown using an umbrella exception that wraps them.  You do need flush(), or can you rely on the user calling commit() (and flush() internally then)?

This enables writing to continue even in the face of errors, but the errors do get reported eventually.  This makes the system to be more robust in the face of transient failure conditions.
               

> StreamingUpdateSolrServer swallows exceptions
> ---------------------------------------------
>
>                 Key: SOLR-3284
>                 URL: https://issues.apache.org/jira/browse/SOLR-3284
>             Project: Solr
>          Issue Type: Improvement
>          Components: clients - java
>    Affects Versions: 3.5, 4.0
>            Reporter: Shawn Heisey
>         Attachments: SOLR-3284.patch
>
>
> StreamingUpdateSolrServer eats exceptions thrown by lower level code, such as HttpClient, when doing adds.  It may happen with other methods, though I know that query and deleteByQuery will throw exceptions.  I believe that this is a result of the queue/Runner design.  That's what makes SUSS perform better, but it means you sacrifice the ability to programmatically determine that there was a problem with your update.  All errors are logged via slf4j, but that's not terribly helpful except with determining what went wrong after the fact.
> When using CommonsHttpSolrServer, I've been able to rely on getting an exception thrown by pretty much any error, letting me use try/catch to detect problems.
> There's probably enough dependent code out there that it would not be a good idea to change the design of SUSS, unless there were alternate constructors or additional methods available to configure new/old behavior.  Fixing this is probably not trivial, so it's probably a better idea to come up with a new server object based on CHSS.  This is outside my current skillset.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] [Comment Edited] (SOLR-3284) StreamingUpdateSolrServer swallows exceptions

Markus Jelsma (Jira)
In reply to this post by Markus Jelsma (Jira)

    [ https://issues.apache.org/jira/browse/SOLR-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13292507#comment-13292507 ]

Mike Sokolov edited comment on SOLR-3284 at 6/10/12 12:14 PM:
--------------------------------------------------------------

Another approach we have been using in a similar parallel writing situation is to have the thread pool maintain a fixed-size list of exceptions.  Whenever a worker throws one, it gets put on the list.  Then at certain barrier conditions (flush, exception list full), the exceptions are collected together and re-thrown using an umbrella exception that wraps them.  At the same time, all worker threads are terminated. You do need flush(), or can you rely on the user calling commit() (and flush() internally then)?

This enables writing to continue even in the face of errors, but the errors do get reported eventually.  This makes the system to be more robust in the face of transient failure conditions.
               
      was (Author: sokolov):
    Another approach we have been using in a similar parallel writing situation is to have the thread pool maintain a fixed-size list of exceptions.  Whenever a worker throws one, it gets put on the list.  Then at certain barrier conditions (flush, exception list full), the exceptions are collected together and re-thrown using an umbrella exception that wraps them.  You do need flush(), or can you rely on the user calling commit() (and flush() internally then)?

This enables writing to continue even in the face of errors, but the errors do get reported eventually.  This makes the system to be more robust in the face of transient failure conditions.
                 

> StreamingUpdateSolrServer swallows exceptions
> ---------------------------------------------
>
>                 Key: SOLR-3284
>                 URL: https://issues.apache.org/jira/browse/SOLR-3284
>             Project: Solr
>          Issue Type: Improvement
>          Components: clients - java
>    Affects Versions: 3.5, 4.0
>            Reporter: Shawn Heisey
>         Attachments: SOLR-3284.patch
>
>
> StreamingUpdateSolrServer eats exceptions thrown by lower level code, such as HttpClient, when doing adds.  It may happen with other methods, though I know that query and deleteByQuery will throw exceptions.  I believe that this is a result of the queue/Runner design.  That's what makes SUSS perform better, but it means you sacrifice the ability to programmatically determine that there was a problem with your update.  All errors are logged via slf4j, but that's not terribly helpful except with determining what went wrong after the fact.
> When using CommonsHttpSolrServer, I've been able to rely on getting an exception thrown by pretty much any error, letting me use try/catch to detect problems.
> There's probably enough dependent code out there that it would not be a good idea to change the design of SUSS, unless there were alternate constructors or additional methods available to configure new/old behavior.  Fixing this is probably not trivial, so it's probably a better idea to come up with a new server object based on CHSS.  This is outside my current skillset.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] [Commented] (SOLR-3284) StreamingUpdateSolrServer swallows exceptions

Markus Jelsma (Jira)
In reply to this post by Markus Jelsma (Jira)

    [ https://issues.apache.org/jira/browse/SOLR-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13292573#comment-13292573 ]

Shawn Heisey commented on SOLR-3284:
------------------------------------

Lance, if your idea (or your idea in combination with Mike's) can be implemented without a ton of work, it should definitely replace my patch.  If it's a royal pain, my patch is there as an interim solution and yours can be handled in another issue.

Is the following a good summary of your idea?  If so, I think your idea is better.  I think it should be the default behavior for 4x and trunk.

A user of ConcurrentUpdateSolrServer would make update requests, then call a publicly exposed flush(), either explicitly or implicitly by calling commit().  If the flush() fails due to a background failure, all requests since the last successful flush() would be rolled back.  If the user code is designed with this in mind, error handling is as good as HttpSolrServer.  HSS (and probably ESS) would use flush() internally.  The user would be free to call flush() themselves, but it would not be required.

I'm curious about how things would be handled with autoCommit or commitWithin.  Is rollback possible when those are used?

               

> StreamingUpdateSolrServer swallows exceptions
> ---------------------------------------------
>
>                 Key: SOLR-3284
>                 URL: https://issues.apache.org/jira/browse/SOLR-3284
>             Project: Solr
>          Issue Type: Improvement
>          Components: clients - java
>    Affects Versions: 3.5, 4.0
>            Reporter: Shawn Heisey
>         Attachments: SOLR-3284.patch
>
>
> StreamingUpdateSolrServer eats exceptions thrown by lower level code, such as HttpClient, when doing adds.  It may happen with other methods, though I know that query and deleteByQuery will throw exceptions.  I believe that this is a result of the queue/Runner design.  That's what makes SUSS perform better, but it means you sacrifice the ability to programmatically determine that there was a problem with your update.  All errors are logged via slf4j, but that's not terribly helpful except with determining what went wrong after the fact.
> When using CommonsHttpSolrServer, I've been able to rely on getting an exception thrown by pretty much any error, letting me use try/catch to detect problems.
> There's probably enough dependent code out there that it would not be a good idea to change the design of SUSS, unless there were alternate constructors or additional methods available to configure new/old behavior.  Fixing this is probably not trivial, so it's probably a better idea to come up with a new server object based on CHSS.  This is outside my current skillset.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] [Commented] (SOLR-3284) StreamingUpdateSolrServer swallows exceptions

Markus Jelsma (Jira)
In reply to this post by Markus Jelsma (Jira)

    [ https://issues.apache.org/jira/browse/SOLR-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13437752#comment-13437752 ]

Alan Woodward commented on SOLR-3284:
-------------------------------------

This interacts really badly with bad documents (SOLR-445) - once one of the Runners passes a bad document to SOLR, all documents subsequently processed by that Runner will be silently discarded, and there doesn't seem to be any way to detect this in client code.
               

> StreamingUpdateSolrServer swallows exceptions
> ---------------------------------------------
>
>                 Key: SOLR-3284
>                 URL: https://issues.apache.org/jira/browse/SOLR-3284
>             Project: Solr
>          Issue Type: Improvement
>          Components: clients - java
>    Affects Versions: 3.5, 4.0-ALPHA
>            Reporter: Shawn Heisey
>         Attachments: SOLR-3284.patch
>
>
> StreamingUpdateSolrServer eats exceptions thrown by lower level code, such as HttpClient, when doing adds.  It may happen with other methods, though I know that query and deleteByQuery will throw exceptions.  I believe that this is a result of the queue/Runner design.  That's what makes SUSS perform better, but it means you sacrifice the ability to programmatically determine that there was a problem with your update.  All errors are logged via slf4j, but that's not terribly helpful except with determining what went wrong after the fact.
> When using CommonsHttpSolrServer, I've been able to rely on getting an exception thrown by pretty much any error, letting me use try/catch to detect problems.
> There's probably enough dependent code out there that it would not be a good idea to change the design of SUSS, unless there were alternate constructors or additional methods available to configure new/old behavior.  Fixing this is probably not trivial, so it's probably a better idea to come up with a new server object based on CHSS.  This is outside my current skillset.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] [Commented] (SOLR-3284) StreamingUpdateSolrServer swallows exceptions

Markus Jelsma (Jira)
In reply to this post by Markus Jelsma (Jira)

    [ https://issues.apache.org/jira/browse/SOLR-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13438072#comment-13438072 ]

Mikhail Khludnev commented on SOLR-3284:
----------------------------------------

Hi,

Excuse me for hijacking, but server-side analog SOLR-3585 is really strict in case of any errors, it gives up and throws them back to clients.
               

> StreamingUpdateSolrServer swallows exceptions
> ---------------------------------------------
>
>                 Key: SOLR-3284
>                 URL: https://issues.apache.org/jira/browse/SOLR-3284
>             Project: Solr
>          Issue Type: Improvement
>          Components: clients - java
>    Affects Versions: 3.5, 4.0-ALPHA
>            Reporter: Shawn Heisey
>         Attachments: SOLR-3284.patch
>
>
> StreamingUpdateSolrServer eats exceptions thrown by lower level code, such as HttpClient, when doing adds.  It may happen with other methods, though I know that query and deleteByQuery will throw exceptions.  I believe that this is a result of the queue/Runner design.  That's what makes SUSS perform better, but it means you sacrifice the ability to programmatically determine that there was a problem with your update.  All errors are logged via slf4j, but that's not terribly helpful except with determining what went wrong after the fact.
> When using CommonsHttpSolrServer, I've been able to rely on getting an exception thrown by pretty much any error, letting me use try/catch to detect problems.
> There's probably enough dependent code out there that it would not be a good idea to change the design of SUSS, unless there were alternate constructors or additional methods available to configure new/old behavior.  Fixing this is probably not trivial, so it's probably a better idea to come up with a new server object based on CHSS.  This is outside my current skillset.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] [Comment Edited] (SOLR-3284) StreamingUpdateSolrServer swallows exceptions

Markus Jelsma (Jira)
In reply to this post by Markus Jelsma (Jira)

    [ https://issues.apache.org/jira/browse/SOLR-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13438072#comment-13438072 ]

Mikhail Khludnev edited comment on SOLR-3284 at 8/21/12 5:37 AM:
-----------------------------------------------------------------

Hi,

Excuse me for hijacking, but server-side analog CUSS SOLR-3585 is really strict in case of any errors, it gives up and throws them back to clients.
               
      was (Author: mkhludnev):
    Hi,

Excuse me for hijacking, but server-side analog SOLR-3585 is really strict in case of any errors, it gives up and throws them back to clients.
                 

> StreamingUpdateSolrServer swallows exceptions
> ---------------------------------------------
>
>                 Key: SOLR-3284
>                 URL: https://issues.apache.org/jira/browse/SOLR-3284
>             Project: Solr
>          Issue Type: Improvement
>          Components: clients - java
>    Affects Versions: 3.5, 4.0-ALPHA
>            Reporter: Shawn Heisey
>         Attachments: SOLR-3284.patch
>
>
> StreamingUpdateSolrServer eats exceptions thrown by lower level code, such as HttpClient, when doing adds.  It may happen with other methods, though I know that query and deleteByQuery will throw exceptions.  I believe that this is a result of the queue/Runner design.  That's what makes SUSS perform better, but it means you sacrifice the ability to programmatically determine that there was a problem with your update.  All errors are logged via slf4j, but that's not terribly helpful except with determining what went wrong after the fact.
> When using CommonsHttpSolrServer, I've been able to rely on getting an exception thrown by pretty much any error, letting me use try/catch to detect problems.
> There's probably enough dependent code out there that it would not be a good idea to change the design of SUSS, unless there were alternate constructors or additional methods available to configure new/old behavior.  Fixing this is probably not trivial, so it's probably a better idea to come up with a new server object based on CHSS.  This is outside my current skillset.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]