Realtime updates delay solrcloud

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Realtime updates delay solrcloud

roySolr
This post was updated on .
Hello Guys,

I want to use the realtime updates mechanism of solrcloud. My setup is as follow:

3 solr engines,
3 zookeeper instances(ensemble)

The setup works great, recovery, leader election etc.

The problem is the realtime updates, it's slow after the servers gets some traffic.

I try to explain it:
I test the realtime update with the following command:

curl <a href="http://SOLRURL:SOLRPORT/solr/update">http://SOLRURL:SOLRPORT/solr/update -H "Content-Type: text/xml" --data-binary '<add><doc><field name="id">3504811</field><field name="website">http://www.google.nl</add></doc>'

I see this in logs of solr server:

Mar 29, 2013 12:38:51 PM org.apache.solr.update.processor.LogUpdateProcessor finish
INFO: [collection1] webapp=/solr path=/update params={} {add=[3504811 (1430841858290876416)]} 0 35


The other solr servers get the following lines in the log:

INFO: [collection1] webapp=/solr path=/update params={distrib.from=<a href="http://SOLRIP:SOLRPORT/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2">http://SOLRIP:SOLRPORT/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[3504811 (1430844456234385408)]} 0 14

This looks good, the doc is added and the leader send this doc to the other solr servers.

First times it takes 1 sec to make the update visible:)

When i send some traffic to the server(200q/s), the update takes +- 30 sec to make it visible.
I stopped the traffic it's still takes 30 sec's to make the update visible. How is it possible? The solrconfig parts:

<autoCommit> 
     <maxTime>600000</maxTime> 
     <openSearcher>false</openSearcher> 
</autoCommit>

<autoSoftCommit> 
    <maxTime>2000</maxTime> 
</autoSoftCommit>


Did i miss something?

Best Regards,
Roy
Reply | Threaded
Open this post in threaded view
|

Re: Realtime updates solrcloud

Erick Erickson
Does the 30 second interval persist for a long time after you stop
your queries? It's possible that your requests are queueing up and you
have a bunch of search in the queue in front of the update....

Best
Erick

On Fri, Mar 29, 2013 at 8:33 AM, roySolr <[hidden email]> wrote:

> Hello Guys,
>
> I want to use the realtime updates mechanism of solrcloud. My setup is as
> follow:
>
> 3 solr engines,
> 3 zookeeper instances(ensemble)
>
> The setup works great, recovery, leader election etc.
>
> The problem is the realtime updates, it's slow after the servers gets some
> traffic.
>
> I try to explain it:
> I test the realtime update with the following command:
>
> *curl <a href="http://SOLRURL:SOLRPORT/solr/update">http://SOLRURL:SOLRPORT/solr/update -H "Content-Type: text/xml"
> --data-binary '<add><doc><field name="id">3504811</field><field
> name="website">http://www.google.nl</add></doc>'*
>
> I see this in logs of solr server:
>
> *Mar 29, 2013 12:38:51 PM
> org.apache.solr.update.processor.LogUpdateProcessor finish
> INFO: [collection1] webapp=/solr path=/update params={} {add=[3504811
> (1430841858290876416)]} 0 35 *
>
> The other solr servers get the following lines in the log:
>
> *INFO: [collection1] webapp=/solr path=/update
> params={distrib.from=<a href="http://SOLRIP:SOLRPORT/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2">http://SOLRIP:SOLRPORT/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2}
> {add=[3504811 (1430844456234385408)]} 0 14*
>
> This looks good, the doc is added and the leader send this doc to the other
> solr servers.
>
> First times it takes 1 sec to make the update visible:)
>
> When i send some traffic to the server(200q/s), the update takes +- 30 sec
> to make it visible.
> I stopped the traffic it's still takes 30 sec's to make the update visible.
> How is it possible? The solrconfig parts:
>
> *<autoCommit>
>      <maxTime>600000</maxTime>
>      <openSearcher>false</openSearcher>
> </autoCommit>
>
> <autoSoftCommit>
>     <maxTime>2000</maxTime>
> </autoSoftCommit>*
>
> Did i miss something?
>
> Best Regards,
> Roy
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Realtime-updates-solrcloud-tp4052370.html
> Sent from the Solr - User mailing list archive at Nabble.com.