OOM at MultiSegmentReader.norms

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

OOM at MultiSegmentReader.norms

vivek sar
Hi,

   I've index of size 50G (around 100 million documents) and growing -
around 2000 records (1 rec = 500 byes) are being written every second
continuously. If I make any search on this index I get OOM. I'm using
default cache settings (512,512,256) in the solrconfig.xml. The search
is using the admin interface (returning 10 rows) with no sorting,
faceting or highlighting. Max heap size is 1024m.

Mar 27, 2009 9:13:41 PM org.apache.solr.common.SolrException log
SEVERE: java.lang.OutOfMemoryError: Java heap space
        at org.apache.lucene.index.MultiSegmentReader.norms(MultiSegmentReader.java:335)
        at org.apache.lucene.search.TermQuery$TermWeight.scorer(TermQuery.java:69)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:132)
        at org.apache.lucene.search.Searcher.search(Searcher.java:126)
        at org.apache.lucene.search.Searcher.search(Searcher.java:105)
        at org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:966)
        at org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:838)
        at org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:269)
        at org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:160)
        at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:169)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:1204)
        at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:303)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:232)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)

What could be the problem?

Thanks,
-vivek
Reply | Threaded
Open this post in threaded view
|

Re: OOM at MultiSegmentReader.norms

Otis Gospodnetic-2

That's a tiny heap.  Part of it is used for indexing, too.  And the fact that your heap is so small shows you are not really making use of that nice ramBufferSizeMB setting. :)

Also, use omitNorms="true" for fields that don't need norms (if their types don't already do that).

Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch



----- Original Message ----

> From: vivek sar <[hidden email]>
> To: [hidden email]
> Sent: Friday, March 27, 2009 6:15:59 PM
> Subject: OOM at MultiSegmentReader.norms
>
> Hi,
>
>    I've index of size 50G (around 100 million documents) and growing -
> around 2000 records (1 rec = 500 byes) are being written every second
> continuously. If I make any search on this index I get OOM. I'm using
> default cache settings (512,512,256) in the solrconfig.xml. The search
> is using the admin interface (returning 10 rows) with no sorting,
> faceting or highlighting. Max heap size is 1024m.
>
> Mar 27, 2009 9:13:41 PM org.apache.solr.common.SolrException log
> SEVERE: java.lang.OutOfMemoryError: Java heap space
>         at
> org.apache.lucene.index.MultiSegmentReader.norms(MultiSegmentReader.java:335)
>         at
> org.apache.lucene.search.TermQuery$TermWeight.scorer(TermQuery.java:69)
>         at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:132)
>         at org.apache.lucene.search.Searcher.search(Searcher.java:126)
>         at org.apache.lucene.search.Searcher.search(Searcher.java:105)
>         at
> org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:966)
>         at
> org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:838)
>         at
> org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:269)
>         at
> org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:160)
>         at
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:169)
>         at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
>         at org.apache.solr.core.SolrCore.execute(SolrCore.java:1204)
>         at
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:303)
>         at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:232)
>         at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
>         at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>         at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
>         at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
>         at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
>         at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
>         at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
>
> What could be the problem?
>
> Thanks,
> -vivek

Reply | Threaded
Open this post in threaded view
|

Re: OOM at MultiSegmentReader.norms

Michael McCandless-2
Still, 1024M ought to be enough to load one field's norms (how many
fields have norms?).  If you do things requiring FieldCache that'll
also consume RAM.

It's also possible you're hitting this bug (false OOME) in Sun's JRE:

  http://issues.apache.org/jira/browse/LUCENE-1566

Feel free to go vote for it!

Mike

On Fri, Mar 27, 2009 at 10:11 PM, Otis Gospodnetic
<[hidden email]> wrote:

>
> That's a tiny heap.  Part of it is used for indexing, too.  And the fact that your heap is so small shows you are not really making use of that nice ramBufferSizeMB setting. :)
>
> Also, use omitNorms="true" for fields that don't need norms (if their types don't already do that).
>
> Otis
> --
> Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
>
>
>
> ----- Original Message ----
>> From: vivek sar <[hidden email]>
>> To: [hidden email]
>> Sent: Friday, March 27, 2009 6:15:59 PM
>> Subject: OOM at MultiSegmentReader.norms
>>
>> Hi,
>>
>>    I've index of size 50G (around 100 million documents) and growing -
>> around 2000 records (1 rec = 500 byes) are being written every second
>> continuously. If I make any search on this index I get OOM. I'm using
>> default cache settings (512,512,256) in the solrconfig.xml. The search
>> is using the admin interface (returning 10 rows) with no sorting,
>> faceting or highlighting. Max heap size is 1024m.
>>
>> Mar 27, 2009 9:13:41 PM org.apache.solr.common.SolrException log
>> SEVERE: java.lang.OutOfMemoryError: Java heap space
>>         at
>> org.apache.lucene.index.MultiSegmentReader.norms(MultiSegmentReader.java:335)
>>         at
>> org.apache.lucene.search.TermQuery$TermWeight.scorer(TermQuery.java:69)
>>         at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:132)
>>         at org.apache.lucene.search.Searcher.search(Searcher.java:126)
>>         at org.apache.lucene.search.Searcher.search(Searcher.java:105)
>>         at
>> org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:966)
>>         at
>> org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:838)
>>         at
>> org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:269)
>>         at
>> org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:160)
>>         at
>> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:169)
>>         at
>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
>>         at org.apache.solr.core.SolrCore.execute(SolrCore.java:1204)
>>         at
>> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:303)
>>         at
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:232)
>>         at
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
>>         at
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>>         at
>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
>>         at
>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
>>         at
>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
>>         at
>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
>>         at
>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
>>
>> What could be the problem?
>>
>> Thanks,
>> -vivek
>
>
Reply | Threaded
Open this post in threaded view
|

Re: OOM at MultiSegmentReader.norms

vivek sar
Thanks Otis and Mike.

I'm indexing total of 9 fields, with 5 having norms turned on. I think
I may not need it and will try use the omitNorms for them.

How do I make use of RAMBuffer in Solr? I couldn't find anything on
this on the Wiki - any pointer?

Thanks,
-vivek

On Sat, Mar 28, 2009 at 1:09 AM, Michael McCandless
<[hidden email]> wrote:

> Still, 1024M ought to be enough to load one field's norms (how many
> fields have norms?).  If you do things requiring FieldCache that'll
> also consume RAM.
>
> It's also possible you're hitting this bug (false OOME) in Sun's JRE:
>
>  http://issues.apache.org/jira/browse/LUCENE-1566
>
> Feel free to go vote for it!
>
> Mike
>
> On Fri, Mar 27, 2009 at 10:11 PM, Otis Gospodnetic
> <[hidden email]> wrote:
>>
>> That's a tiny heap.  Part of it is used for indexing, too.  And the fact that your heap is so small shows you are not really making use of that nice ramBufferSizeMB setting. :)
>>
>> Also, use omitNorms="true" for fields that don't need norms (if their types don't already do that).
>>
>> Otis
>> --
>> Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
>>
>>
>>
>> ----- Original Message ----
>>> From: vivek sar <[hidden email]>
>>> To: [hidden email]
>>> Sent: Friday, March 27, 2009 6:15:59 PM
>>> Subject: OOM at MultiSegmentReader.norms
>>>
>>> Hi,
>>>
>>>    I've index of size 50G (around 100 million documents) and growing -
>>> around 2000 records (1 rec = 500 byes) are being written every second
>>> continuously. If I make any search on this index I get OOM. I'm using
>>> default cache settings (512,512,256) in the solrconfig.xml. The search
>>> is using the admin interface (returning 10 rows) with no sorting,
>>> faceting or highlighting. Max heap size is 1024m.
>>>
>>> Mar 27, 2009 9:13:41 PM org.apache.solr.common.SolrException log
>>> SEVERE: java.lang.OutOfMemoryError: Java heap space
>>>         at
>>> org.apache.lucene.index.MultiSegmentReader.norms(MultiSegmentReader.java:335)
>>>         at
>>> org.apache.lucene.search.TermQuery$TermWeight.scorer(TermQuery.java:69)
>>>         at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:132)
>>>         at org.apache.lucene.search.Searcher.search(Searcher.java:126)
>>>         at org.apache.lucene.search.Searcher.search(Searcher.java:105)
>>>         at
>>> org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:966)
>>>         at
>>> org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:838)
>>>         at
>>> org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:269)
>>>         at
>>> org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:160)
>>>         at
>>> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:169)
>>>         at
>>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
>>>         at org.apache.solr.core.SolrCore.execute(SolrCore.java:1204)
>>>         at
>>> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:303)
>>>         at
>>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:232)
>>>         at
>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
>>>         at
>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>>>         at
>>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
>>>         at
>>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
>>>         at
>>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
>>>         at
>>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
>>>         at
>>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
>>>
>>> What could be the problem?
>>>
>>> Thanks,
>>> -vivek
>>
>>
>
Reply | Threaded
Open this post in threaded view
|

Re: OOM at MultiSegmentReader.norms

Yonik Seeley-2-2
On Mon, Mar 30, 2009 at 12:53 PM, vivek sar <[hidden email]> wrote:
> I'm indexing total of 9 fields, with 5 having norms turned on.

So that's 500MB for norms alone, plus memory for Lucene's term index
(every 128th term by default).  Solr also opens a new
IndexReader/Searcher before closing the old one, so there are no
"outages"... a period when the old searcher has been closed but a new
searcher isn't yet "ready".


-Yonik
http://www.lucidimagination.com
Reply | Threaded
Open this post in threaded view
|

Re: OOM at MultiSegmentReader.norms

Otis Gospodnetic-2
In reply to this post by vivek sar

Vivek,
No need for any RAMBuffer stuff, just use the appropriate ramBufferSizeMB value in solrconfig.xml.  This will only help with indexing, though!


Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch



----- Original Message ----

> From: vivek sar <[hidden email]>
> To: [hidden email]
> Sent: Monday, March 30, 2009 12:53:21 PM
> Subject: Re: OOM at MultiSegmentReader.norms
>
> Thanks Otis and Mike.
>
> I'm indexing total of 9 fields, with 5 having norms turned on. I think
> I may not need it and will try use the omitNorms for them.
>
> How do I make use of RAMBuffer in Solr? I couldn't find anything on
> this on the Wiki - any pointer?
>
> Thanks,
> -vivek
>
> On Sat, Mar 28, 2009 at 1:09 AM, Michael McCandless
> wrote:
> > Still, 1024M ought to be enough to load one field's norms (how many
> > fields have norms?).  If you do things requiring FieldCache that'll
> > also consume RAM.
> >
> > It's also possible you're hitting this bug (false OOME) in Sun's JRE:
> >
> >  http://issues.apache.org/jira/browse/LUCENE-1566
> >
> > Feel free to go vote for it!
> >
> > Mike
> >
> > On Fri, Mar 27, 2009 at 10:11 PM, Otis Gospodnetic
> > wrote:
> >>
> >> That's a tiny heap.  Part of it is used for indexing, too.  And the fact that
> your heap is so small shows you are not really making use of that nice
> ramBufferSizeMB setting. :)
> >>
> >> Also, use omitNorms="true" for fields that don't need norms (if their types
> don't already do that).
> >>
> >> Otis
> >> --
> >> Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
> >>
> >>
> >>
> >> ----- Original Message ----
> >>> From: vivek sar
> >>> To: [hidden email]
> >>> Sent: Friday, March 27, 2009 6:15:59 PM
> >>> Subject: OOM at MultiSegmentReader.norms
> >>>
> >>> Hi,
> >>>
> >>>    I've index of size 50G (around 100 million documents) and growing -
> >>> around 2000 records (1 rec = 500 byes) are being written every second
> >>> continuously. If I make any search on this index I get OOM. I'm using
> >>> default cache settings (512,512,256) in the solrconfig.xml. The search
> >>> is using the admin interface (returning 10 rows) with no sorting,
> >>> faceting or highlighting. Max heap size is 1024m.
> >>>
> >>> Mar 27, 2009 9:13:41 PM org.apache.solr.common.SolrException log
> >>> SEVERE: java.lang.OutOfMemoryError: Java heap space
> >>>         at
> >>>
> org.apache.lucene.index.MultiSegmentReader.norms(MultiSegmentReader.java:335)
> >>>         at
> >>> org.apache.lucene.search.TermQuery$TermWeight.scorer(TermQuery.java:69)
> >>>         at
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:132)
> >>>         at org.apache.lucene.search.Searcher.search(Searcher.java:126)
> >>>         at org.apache.lucene.search.Searcher.search(Searcher.java:105)
> >>>         at
> >>>
> org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:966)
> >>>         at
> >>>
> org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:838)
> >>>         at
> >>> org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:269)
> >>>         at
> >>>
> org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:160)
> >>>         at
> >>>
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:169)
> >>>         at
> >>>
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
> >>>         at org.apache.solr.core.SolrCore.execute(SolrCore.java:1204)
> >>>         at
> >>>
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:303)
> >>>         at
> >>>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:232)
> >>>         at
> >>>
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
> >>>         at
> >>>
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> >>>         at
> >>>
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
> >>>         at
> >>>
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
> >>>         at
> >>>
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
> >>>         at
> >>>
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
> >>>         at
> >>>
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
> >>>
> >>> What could be the problem?
> >>>
> >>> Thanks,
> >>> -vivek
> >>
> >>
> >