latency in solr response is observed after index is updated

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

latency in solr response is observed after index is updated

bharath venkatesh-3
Hi,

     We are observing latency (some times huge latency upto 10-20 secs)
in solr response  after index is updated . whats the reason of this
latency and how can it be minimized ?

Note: our index size is pretty large.

any help would be appreciated as we largely affected by it

Thanks in advance.
Bharath
This message is intended only for the use of the addressee and may contain information that is privileged, confidential
and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the
employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any
dissemination, distribution or copying of this communication is strictly prohibited. If you have received this e-mail
in error, please notify us immediately by return e-mail and delete this e-mail and all attachments from your system.

Reply | Threaded
Open this post in threaded view
|

Re: latency in solr response is observed after index is updated

kalidoss-2
r u using solr-config for committing the document?

bharath venkatesh wrote:

> Hi,
>
>     We are observing latency (some times huge latency upto 10-20 secs)
> in solr response  after index is updated . whats the reason of this
> latency and how can it be minimized ?
> Note: our index size is pretty large.
>
> any help would be appreciated as we largely affected by it
>
> Thanks in advance.
> Bharath
> This message is intended only for the use of the addressee and may
> contain information that is privileged, confidential and exempt from
> disclosure under applicable law. If the reader of this message is not
> the intended recipient, or the employee or agent responsible for
> delivering the message to the intended recipient, you are hereby
> notified that any dissemination, distribution or copying of this
> communication is strictly prohibited. If you have received this e-mail
> in error, please notify us immediately by return e-mail and delete
> this e-mail and all attachments from your system.
>
>



Get your world in your inbox!

Mail, widgets, documents, spreadsheets, organizer and much more with your Sifymail WIYI id!
Log on to http://www.sify.com

********** DISCLAIMER **********
Information contained and transmitted by this E-MAIL is proprietary to
Sify Limited and is intended for use only by the individual or entity to
which it is addressed, and may contain information that is privileged,
confidential or exempt from disclosure under applicable law. If this is a
forwarded message, the content of this E-MAIL may not have been sent with
the authority of the Company. If you are not the intended recipient, an
agent of the intended recipient or a  person responsible for delivering the
information to the named recipient,  you are notified that any use,
distribution, transmission, printing, copying or dissemination of this
information in any way or in any manner is strictly prohibited. If you have
received this communication in error, please delete this mail & notify us
immediately at [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: latency in solr response is observed after index is updated

bharath venkatesh-3
Hi Kalidoss,
 
   I am not aware of using solr-config for committing the document .
but I have mentioned below how we update and  commit documents:
 
curl http://solr_url/update --data-binary @feeds.xml -H
'Content-type:text/xml; charset=utf-8'
curl http://solr_url/update --data-binary '<commit/>' -H
'Content-type:text/xml; charset=utf-8'

where feeds.xml contains the document in xml format

we have master and slave replication for solr server.

updates happens in master , snappuller and snapinstaller is run on
slaves periodically
queries don't happen at master , only happens at slaves

is there any thing which can be said from above information ?

Thanks,
Bharath



-----Original Message-----
From: kalidoss [mailto:[hidden email]]
Sent: Tue 12/1/2009 2:38 PM
To: [hidden email]
Subject: Re: latency in solr response  is observed  after index is updated
 
r u using solr-config for committing the document?

bharath venkatesh wrote:

> Hi,
>
>     We are observing latency (some times huge latency upto 10-20 secs)
> in solr response  after index is updated . whats the reason of this
> latency and how can it be minimized ?
> Note: our index size is pretty large.
>
> any help would be appreciated as we largely affected by it
>
> Thanks in advance.
> Bharath





This message is intended only for the use of the addressee and may contain information that is privileged, confidential
and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the
employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any
dissemination, distribution or copying of this communication is strictly prohibited. If you have received this e-mail
in error, please notify us immediately by return e-mail and delete this e-mail and all attachments from your system.
Reply | Threaded
Open this post in threaded view
|

Re: latency in solr response is observed after index is updated

Kay Kay-2-3
What would be the average doc size.  What is the autoCommit frequency
set in solrconfig.xml .

Another place to look at is the field cache size and the nature of
warmup queries run after a new searcher is created ( happens due to a
commit ).



Bharath Venkatesh wrote:

> Hi Kalidoss,
>  
>    I am not aware of using solr-config for committing the document .
> but I have mentioned below how we update and  commit documents:
>  
> curl http://solr_url/update --data-binary @feeds.xml -H
> 'Content-type:text/xml; charset=utf-8'
> curl http://solr_url/update --data-binary '<commit/>' -H
> 'Content-type:text/xml; charset=utf-8'
>
> where feeds.xml contains the document in xml format
>
> we have master and slave replication for solr server.
>
> updates happens in master , snappuller and snapinstaller is run on
> slaves periodically
> queries don't happen at master , only happens at slaves
>
> is there any thing which can be said from above information ?
>
> Thanks,
> Bharath
>
>
>
> -----Original Message-----
> From: kalidoss [mailto:[hidden email]]
> Sent: Tue 12/1/2009 2:38 PM
> To: [hidden email]
> Subject: Re: latency in solr response  is observed  after index is updated
>  
> r u using solr-config for committing the document?
>
> bharath venkatesh wrote:
>  
>> Hi,
>>
>>     We are observing latency (some times huge latency upto 10-20 secs)
>> in solr response  after index is updated . whats the reason of this
>> latency and how can it be minimized ?
>> Note: our index size is pretty large.
>>
>> any help would be appreciated as we largely affected by it
>>
>> Thanks in advance.
>> Bharath
>>    
>
>
>
>
>
>
> This message is intended only for the use of the addressee and may contain information that is privileged, confidential
> and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the
> employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any
> dissemination, distribution or copying of this communication is strictly prohibited. If you have received this e-mail
> in error, please notify us immediately by return e-mail and delete this e-mail and all attachments from your system.
>  

Reply | Threaded
Open this post in threaded view
|

Re: latency in solr response is observed after index is updated

bharath venkatesh-3
In reply to this post by bharath venkatesh-3

Hi Kay Kay ,
     We have commented out  auto commit frequency in solrconfig.xml

below is the cache configuration:-

    <filterCache
      class="solr.LRUCache"
      size="512"
      initialSize="512"
      autowarmCount="256"/>

   <!-- queryResultCache caches results of searches - ordered lists of
         document ids (DocList) based on a query, a sort, and the range
         of documents requested.  -->
    <queryResultCache
      class="solr.LRUCache"
      size="512"
      initialSize="512"
      autowarmCount="256"/>

  <!-- documentCache caches Lucene Document objects (the stored fields for each document).
       Since Lucene internal document ids are transient, this cache will not be autowarmed.  -->
    <documentCache
      class="solr.LRUCache"
      size="512"
      initialSize="512"
      autowarmCount="0"/>

will further requests after index is updated  wait for auto warming to complete ?

Thanks,
Bharath


Kay Kay wrote:

> What would be the average doc size.  What is the autoCommit frequency set in solrconfig.xml .
>
> Another place to look at is the field cache size and the nature of warmup queries run after a new searcher is created ( happens due to a commit ).
>
>
>
> Bharath Venkatesh wrote:
>> Hi Kalidoss,
>>      I am not aware of using solr-config for committing the document . but I have mentioned below how we update and  commit documents:
>>  
>> curl http://solr_url/update --data-binary @feeds.xml -H 'Content-type:text/xml; charset=utf-8'
>> curl http://solr_url/update --data-binary '<commit/>' -H 'Content-type:text/xml; charset=utf-8'
>>
>> where feeds.xml contains the document in xml format
>>
>> we have master and slave replication for solr server.
>>
>> updates happens in master , snappuller and snapinstaller is run on slaves periodically
>> queries don't happen at master , only happens at slaves
>>
>> is there any thing which can be said from above information ?
>>
>> Thanks,
>> Bharath
>>
>>
>>
>> -----Original Message-----
>> From: kalidoss [mailto:[hidden email]]
>> Sent: Tue 12/1/2009 2:38 PM
>> To: [hidden email]
>> Subject: Re: latency in solr response  is observed  after index is updated
>>  
>> r u using solr-config for committing the document?
>>
>> bharath venkatesh wrote:
>>  
>>> Hi,
>>>
>>>     We are observing latency (some times huge latency upto 10-20 secs) in solr response  after index is updated . whats the reason of this latency and how can it be minimized ?
>>> Note: our index size is pretty large.
>>>
>>> any help would be appreciated as we largely affected by it
>>>
>>> Thanks in advance.
>>> Bharath
>>>    

This message is intended only for the use of the addressee and may contain information that is privileged, confidential
and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the
employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any
dissemination, distribution or copying of this communication is strictly prohibited. If you have received this e-mail
in error, please notify us immediately by return e-mail and delete this e-mail and all attachments from your system.
Reply | Threaded
Open this post in threaded view
|

Re: latency in solr response is observed after index is updated

hossman
In reply to this post by bharath venkatesh-3

:     We are observing latency (some times huge latency upto 10-20 secs) in solr
: response  after index is updated . whats the reason of this latency and how
: can it be minimized ?
: Note: our index size is pretty large.

Please read the following wiki pages...

http://wiki.apache.org/solr/SolrPerformanceFactors
http://wiki.apache.org/solr/SolrCaching

...and if you still have any additional questions, please provide the
relevant information about your setup (ie: solrconfig.xml, frequency of
running snapinstaller, examples of the exact queries that are slow,
etc...)


-Hoss