SolrCloud cache warming issues

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

SolrCloud cache warming issues

Markus Jelsma-2
Hi,

We have about 500 top queries for each core in our test cluster. They are routed to the default request handler and are not distributed, facetting is enabled and most common filters are present and rows not specified but the default is 10. After server start up we only see a few items being added to the filterCache and some added to the fieldValueCache and fieldCache, which is good. The queryResultCache is disabled and the documentCache is not being populated for some unknown reason.

Why would the documentCache not be populated via firstSearcher warming queries with a non-zero value for rows?

Thanks,
Markus
Reply | Threaded
Open this post in threaded view
|

Re: SolrCloud cache warming issues

Yonik Seeley-2-2
On Tue, Jun 26, 2012 at 6:53 AM, Markus Jelsma
<[hidden email]> wrote:
> Why would the documentCache not be populated via firstSearcher warming queries with a non-zero value for rows?

Solr streams documents (the stored fields) returned to the user (so
very large result sets can be supported w/o having the whole thing in
memory).
A warming query finds the document ids matching a query, but does not
send them anywhere (and the stored fields aren't needed for anything
else), hence the stored fields are never loaded.

-Yonik
http://lucidimagination.com
Reply | Threaded
Open this post in threaded view
|

Re: SolrCloud cache warming issues

Erik Hatcher-4

On Jun 27, 2012, at 12:01 , Yonik Seeley wrote:

> On Tue, Jun 26, 2012 at 6:53 AM, Markus Jelsma
> <[hidden email]> wrote:
>> Why would the documentCache not be populated via firstSearcher warming queries with a non-zero value for rows?
>
> Solr streams documents (the stored fields) returned to the user (so
> very large result sets can be supported w/o having the whole thing in
> memory).
> A warming query finds the document ids matching a query, but does not
> send them anywhere (and the stored fields aren't needed for anything
> else), hence the stored fields are never loaded.


But if highlighting were enabled on those warming queries, it'd fill in the document cache, right?

        Erik

Reply | Threaded
Open this post in threaded view
|

Re: SolrCloud cache warming issues

Yonik Seeley-2-2
On Wed, Jun 27, 2012 at 12:23 PM, Erik Hatcher <[hidden email]> wrote:

>
> On Jun 27, 2012, at 12:01 , Yonik Seeley wrote:
>
>> On Tue, Jun 26, 2012 at 6:53 AM, Markus Jelsma
>> <[hidden email]> wrote:
>>> Why would the documentCache not be populated via firstSearcher warming queries with a non-zero value for rows?
>>
>> Solr streams documents (the stored fields) returned to the user (so
>> very large result sets can be supported w/o having the whole thing in
>> memory).
>> A warming query finds the document ids matching a query, but does not
>> send them anywhere (and the stored fields aren't needed for anything
>> else), hence the stored fields are never loaded.
>
>
> But if highlighting were enabled on those warming queries, it'd fill in the document cache, right?

Correct.

-Yonik
http://lucidimagination.com
Reply | Threaded
Open this post in threaded view
|

RE: SolrCloud cache warming issues

Markus Jelsma-2
Interesting!

We also tried routing the warming queries through our main search request handler, with highlighting enabled, that has distrib=true as default. To prevent warming queries to run over the cluster on all instances we set distrib=false in the warming queries. The queries were fired at start up but the Solr instance stays unreachable from the outside. It caused an awful amount of socket time out exceptions.

How is warming on a cluster supposed to behave? Is distrib=false enforced if it is a default for the used handler?

Thanks

 
 
-----Original message-----

> From:Yonik Seeley <[hidden email]>
> Sent: Wed 27-Jun-2012 18:27
> To: [hidden email]
> Subject: Re: SolrCloud cache warming issues
>
> On Wed, Jun 27, 2012 at 12:23 PM, Erik Hatcher <[hidden email]> wrote:
> >
> > On Jun 27, 2012, at 12:01 , Yonik Seeley wrote:
> >
> >> On Tue, Jun 26, 2012 at 6:53 AM, Markus Jelsma
> >> <[hidden email]> wrote:
> >>> Why would the documentCache not be populated via firstSearcher warming queries with a non-zero value for rows?
> >>
> >> Solr streams documents (the stored fields) returned to the user (so
> >> very large result sets can be supported w/o having the whole thing in
> >> memory).
> >> A warming query finds the document ids matching a query, but does not
> >> send them anywhere (and the stored fields aren't needed for anything
> >> else), hence the stored fields are never loaded.
> >
> >
> > But if highlighting were enabled on those warming queries, it'd fill in the document cache, right?
>
> Correct.
>
> -Yonik
> http://lucidimagination.com
>