Synchronizing commit and optimize

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Synchronizing commit and optimize

Marcus Stratmann
Hello,

when doing a commit or optimize the operation takes quite long (in my
test case at least some minutes). When I submit the command via curl, I
get the response "curl: (52) Empty reply from server" though solr is
still working (as I can see from the process list and the admin
interface). I tried the options "--connect-timeout" and "--max-time"
but still curl returns after some seconds though the request is not
fully processed. The same thing happens when I submit the commands from
a PHP-script (ensuring that it waits for a server response).

I'm not sure if I'm doing something wrong, but I could imagine three
causes for this.
1) curl (or my script) simply doesn't wail long enough to get a
response from the server. Well, I think I've ensured that this is not
the case, see above.
2) Jetty (I'm using the standard installation from the example) doesn't
wait long enough to get a response from Solr and thus returns an empty
response.
3) Solr itself is the problem.

For me point 2 sounds reasonable but I have no idea how to test this.

I'm also getting empty responses when adding documents to the index.
This happens every time when a multiple of one million documents have
been added to the index. I guess the reason is, that I have a merge
factor of 10 and that the operation of adding a document takes longer
when a multiple of 10^6 documents is reached.

Is there any way to synchronize a commit or optimize with other
commands (for example in a shell script)? The example in the script
"commit" in src/scripts doesn't use any special arguments with curl and
returns some seconds after submiting the request, so this doesn't seem
to work.

Thanks in advance,
Marcus

--
Echte DSL-Flatrate dauerhaft für 0,- Euro*!
"Feel free" mit GMX DSL! http://www.gmx.net/de/go/dsl
Reply | Threaded
Open this post in threaded view
|

Re: Synchronizing commit and optimize

Yonik Seeley
On 4/24/06, Marcus Stratmann <[hidden email]> wrote:
> When I submit the command via curl, I
> get the response "curl: (52) Empty reply from server" though solr is
> still working

I think you are probably right about Jetty timing out the request.
Solr doesn't implement timeouts for requests, and I havent' seen this
behavior with Solr running on Resin.

You could try another app server like Tomcat, or perhaps figure out of
the Jetty timeout is configurable.

-Yonik
Reply | Threaded
Open this post in threaded view
|

Re: Synchronizing commit and optimize

Marcus Stratmann
Yonik Seeley wrote:
>I think you are probably right about Jetty timing out the request.
>Solr doesn't implement timeouts for requests, and I havent' seen this
>behavior with Solr running on Resin.
>
>You could try another app server like Tomcat, or perhaps figure out of
>the Jetty timeout is configurable.

You were right, it's an Jetty issue.
In Jetty's configuration in jetty.xml I changed the parameter
maxIdleTime which seems to be in milliseconds (I wasn't able to
find documentation for this anywhere). Increasing this value to
3600000 (1 hour) did the trick for me. The line is
<Set name="maxIdleTime">3600000</Set>

The default value in the example installation is 30000. Maybe
it wolud be a good idea to increase this, too.

Thanks,
Marcus

Reply | Threaded
Open this post in threaded view
|

Re: Synchronizing commit and optimize

Chris Hostetter-3

: The default value in the example installation is 30000. Maybe
: it wolud be a good idea to increase this, too.

The example is really just designed to make it easy to see Solr in action
really fast ... not to demonstrate any recommended configuration of the
Jetty -- if we make too many tweaks to the Jetty config, people might
missunderstand.

But we should definitely add something to the FAQ about timeouts, i'll do
that now.


-Hoss