Question about Lucene FieldCache

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Question about Lucene FieldCache

Yago Riveiro
Hi,

After some reading into the documentation, supposedly the Lucene FieldCache is the only one that it's not possible to disable.

Fetching the config for a collection through the REST API I found an entry like this:

"query": {
"useFilterForSortedQuery": true,
"queryResultWindowSize": 1,
"queryResultMaxDocsCached": 0,
"enableLazyFieldLoading": true,
"maxBooleanClauses": 8192,
"": {
"size": "10000",
"showItems": "-1",
"initialSize": "10",
"name": "fieldValueCache"
}
},

My questions:

- That size, 10000 is for all files of the collection schema or is 10000 for each field defined?
- If I reload the collection the caches are wiped?

Regards,

/Yago
Best regards /Yago
Reply | Threaded
Open this post in threaded view
|

Re: Question about Lucene FieldCache

Mikhail Khludnev-2
Hello, Yago.

"size": "10000", "showItems": "-1", "initialSize": "10", "name":
"fieldValueCache"

These are Solr's UnInvertedFields, not Lucene's FieldCache.
That 10000 is for all fields of the collection schema.
Collection reload or commit drop all entries from this cache.


On Mon, Jan 9, 2017 at 1:30 PM, Yago Riveiro <[hidden email]> wrote:

> Hi,
>
> After some reading into the documentation, supposedly the Lucene FieldCache
> is the only one that it's not possible to disable.
>
> Fetching the config for a collection through the REST API I found an entry
> like this:
>
> "query": {
> "useFilterForSortedQuery": true,
> "queryResultWindowSize": 1,
> "queryResultMaxDocsCached": 0,
> "enableLazyFieldLoading": true,
> "maxBooleanClauses": 8192,
> "": {
> "size": "10000",
> "showItems": "-1",
> "initialSize": "10",
> "name": "fieldValueCache"
> }
> },
>
> My questions:
>
> - That size, 10000 is for all files of the collection schema or is 10000
> for
> each field defined?
> - If I reload the collection the caches are wiped?
>
> Regards,
>
> /Yago
>
>
>
> -----
> Best regards
>
> /Yago
> --
> View this message in context: http://lucene.472066.n3.
> nabble.com/Question-about-Lucene-FieldCache-tp4313062.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>



--
Sincerely yours
Mikhail Khludnev
Reply | Threaded
Open this post in threaded view
|

Re: Question about Lucene FieldCache

Yago Riveiro
Thanks for re reply Mikhail,

Do you know if the 10000 value is configurable? My insert rate is so high (5000 docs/s) that the cache it's quite useless.

In the case of the Lucene field cache, it's possible "clean" it in some way?

Some cache is eating my memory heap.
Best regards /Yago
Reply | Threaded
Open this post in threaded view
|

Re: Question about Lucene FieldCache

Mikhail Khludnev-2
On Mon, Jan 9, 2017 at 2:17 PM, Yago Riveiro <[hidden email]> wrote:

> Thanks for re reply Mikhail,
>
> Do you know if the 10000 value is configurable?

yes. in solrconfig.xml
https://cwiki.apache.org/confluence/display/solr/Query+Settings+in+SolrConfig#QuerySettingsinSolrConfig-Caches
iirc you cant' fully disable it setting size to 0.


> My insert rate is so high
> (5000 docs/s) that the cache it's quite useless.
>
> In the case of the Lucene field cache, it's possible "clean" it in some
> way?
>
> Even it would be possible, the first sorting query or so loads it back.

> Some cache is eating my memory heap.
>
Probably you need to dedicate master which won't load FieldCache.


>
>
>
> -----
> Best regards
>
> /Yago
> --
> View this message in context: http://lucene.472066.n3.
> nabble.com/Question-about-Lucene-FieldCache-tp4313062p4313069.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>



--
Sincerely yours
Mikhail Khludnev
Reply | Threaded
Open this post in threaded view
|

Re: Question about Lucene FieldCache

Yago Riveiro
The documentation says that the only caches configurable are:

- filterCache
- queryResultCache
- documentCache
- user defined caches

There is no entry for fieldValueCache and in my case all of list in the documentation are disable ...

--

/Yago Riveiro

On 9 Jan 2017 13:20 +0000, Mikhail Khludnev <[hidden email]>, wrote:

> On Mon, Jan 9, 2017 at 2:17 PM, Yago Riveiro <[hidden email]> wrote:
>
> > Thanks for re reply Mikhail,
> >
> > Do you know if the 10000 value is configurable?
>
> yes. in solrconfig.xml
> https://cwiki.apache.org/confluence/display/solr/Query+Settings+in+SolrConfig#QuerySettingsinSolrConfig-Caches
> iirc you cant' fully disable it setting size to 0.
>
>
> > My insert rate is so high
> > (5000 docs/s) that the cache it's quite useless.
> >
> > In the case of the Lucene field cache, it's possible "clean" it in some
> > way?
> >
> > Even it would be possible, the first sorting query or so loads it back.
>
> > Some cache is eating my memory heap.
> >
> Probably you need to dedicate master which won't load FieldCache.
>
>
> >
> >
> >
> > -----
> > Best regards
> >
> > /Yago
> > --
> > View this message in context: http://lucene.472066.n3.
> > nabble.com/Question-about-Lucene-FieldCache-tp4313062p4313069.html
> > Sent from the Solr - User mailing list archive at Nabble.com.
> >
>
>
>
> --
> Sincerely yours
> Mikhail Khludnev
Best regards /Yago
Reply | Threaded
Open this post in threaded view
|

Re: Question about Lucene FieldCache

Billnbell
Try disabling and perf may get better

Bill Bell
Sent from mobile


> On Jan 9, 2017, at 6:41 AM, Yago Riveiro <[hidden email]> wrote:
>
> The documentation says that the only caches configurable are:
>
> - filterCache
> - queryResultCache
> - documentCache
> - user defined caches
>
> There is no entry for fieldValueCache and in my case all of list in the documentation are disable ...
>
> --
>
> /Yago Riveiro
>
>> On 9 Jan 2017 13:20 +0000, Mikhail Khludnev <[hidden email]>, wrote:
>>> On Mon, Jan 9, 2017 at 2:17 PM, Yago Riveiro <[hidden email]> wrote:
>>>
>>> Thanks for re reply Mikhail,
>>>
>>> Do you know if the 10000 value is configurable?
>>
>> yes. in solrconfig.xml
>> https://cwiki.apache.org/confluence/display/solr/Query+Settings+in+SolrConfig#QuerySettingsinSolrConfig-Caches
>> iirc you cant' fully disable it setting size to 0.
>>
>>
>>> My insert rate is so high
>>> (5000 docs/s) that the cache it's quite useless.
>>>
>>> In the case of the Lucene field cache, it's possible "clean" it in some
>>> way?
>>>
>>> Even it would be possible, the first sorting query or so loads it back.
>>
>>> Some cache is eating my memory heap.
>>>
>> Probably you need to dedicate master which won't load FieldCache.
>>
>>
>>>
>>>
>>>
>>> -----
>>> Best regards
>>>
>>> /Yago
>>> --
>>> View this message in context: http://lucene.472066.n3.
>>> nabble.com/Question-about-Lucene-FieldCache-tp4313062p4313069.html
>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>
>>
>>
>>
>> --
>> Sincerely yours
>> Mikhail Khludnev
Reply | Threaded
Open this post in threaded view
|

Re: Question about Lucene FieldCache

Mikhail Khludnev-2
In reply to this post by Yago Riveiro
This probably says why
https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/core/SolrConfig.java#L258

On Mon, Jan 9, 2017 at 4:41 PM, Yago Riveiro <[hidden email]> wrote:

> The documentation says that the only caches configurable are:
>
> - filterCache
> - queryResultCache
> - documentCache
> - user defined caches
>
> There is no entry for fieldValueCache and in my case all of list in the
> documentation are disable ...
>
> --
>
> /Yago Riveiro
>
> On 9 Jan 2017 13:20 +0000, Mikhail Khludnev <[hidden email]>, wrote:
> > On Mon, Jan 9, 2017 at 2:17 PM, Yago Riveiro <[hidden email]>
> wrote:
> >
> > > Thanks for re reply Mikhail,
> > >
> > > Do you know if the 10000 value is configurable?
> >
> > yes. in solrconfig.xml
> > https://cwiki.apache.org/confluence/display/solr/Query+
> Settings+in+SolrConfig#QuerySettingsinSolrConfig-Caches
> > iirc you cant' fully disable it setting size to 0.
> >
> >
> > > My insert rate is so high
> > > (5000 docs/s) that the cache it's quite useless.
> > >
> > > In the case of the Lucene field cache, it's possible "clean" it in some
> > > way?
> > >
> > > Even it would be possible, the first sorting query or so loads it back.
> >
> > > Some cache is eating my memory heap.
> > >
> > Probably you need to dedicate master which won't load FieldCache.
> >
> >
> > >
> > >
> > >
> > > -----
> > > Best regards
> > >
> > > /Yago
> > > --
> > > View this message in context: http://lucene.472066.n3.
> > > nabble.com/Question-about-Lucene-FieldCache-tp4313062p4313069.html
> > > Sent from the Solr - User mailing list archive at Nabble.com.
> > >
> >
> >
> >
> > --
> > Sincerely yours
> > Mikhail Khludnev
>



--
Sincerely yours
Mikhail Khludnev
Reply | Threaded
Open this post in threaded view
|

Re: Question about Lucene FieldCache

Yago Riveiro
Ok, then I need to configure to reduce the size of the cache.

Thanks for the help Mikhail.

--

/Yago Riveiro

On 9 Jan 2017 17:01 +0000, Mikhail Khludnev <[hidden email]>, wrote:

> This probably says why
> https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/core/SolrConfig.java#L258
>
> On Mon, Jan 9, 2017 at 4:41 PM, Yago Riveiro <[hidden email]> wrote:
>
> > The documentation says that the only caches configurable are:
> >
> > - filterCache
> > - queryResultCache
> > - documentCache
> > - user defined caches
> >
> > There is no entry for fieldValueCache and in my case all of list in the
> > documentation are disable ...
> >
> > --
> >
> > /Yago Riveiro
> >
> > On 9 Jan 2017 13:20 +0000, Mikhail Khludnev <[hidden email]>, wrote:
> > > On Mon, Jan 9, 2017 at 2:17 PM, Yago Riveiro <[hidden email]
> > wrote:
> > >
> > > > Thanks for re reply Mikhail,
> > > >
> > > > Do you know if the 10000 value is configurable?
> > >
> > > yes. in solrconfig.xml
> > > https://cwiki.apache.org/confluence/display/solr/Query+
> > Settings+in+SolrConfig#QuerySettingsinSolrConfig-Caches
> > > iirc you cant' fully disable it setting size to 0.
> > >
> > >
> > > > My insert rate is so high
> > > > (5000 docs/s) that the cache it's quite useless.
> > > >
> > > > In the case of the Lucene field cache, it's possible "clean" it in some
> > > > way?
> > > >
> > > > Even it would be possible, the first sorting query or so loads it back.
> > >
> > > > Some cache is eating my memory heap.
> > > >
> > > Probably you need to dedicate master which won't load FieldCache.
> > >
> > >
> > > >
> > > >
> > > >
> > > > -----
> > > > Best regards
> > > >
> > > > /Yago
> > > > --
> > > > View this message in context: http://lucene.472066.n3.
> > > > nabble.com/Question-about-Lucene-FieldCache-tp4313062p4313069.html
> > > > Sent from the Solr - User mailing list archive at Nabble.com.
> > > >
> > >
> > >
> > >
> > > --
> > > Sincerely yours
> > > Mikhail Khludnev
> >
>
>
>
> --
> Sincerely yours
> Mikhail Khludnev
Best regards /Yago