Solr OpenBitSet OutofMemory Error

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Solr OpenBitSet OutofMemory Error

harish.agarwal
Hello,
After running a nightly release from around January of Solr for about 4 weeks without any problems, I'm starting to see OutofMemory errors:

Mar 24, 2009 1:35:36 AM org.apache.solr.common.SolrException log
SEVERE: java.lang.OutOfMemoryError: Java heap space
        at org.apache.lucene.util.OpenBitSet.clone(OpenBitSet.java:640)


Is this a common error to see?  I'm running a lot of faceted queries on an index with about 7.5 million documents.  I'm giving about 8GBs of memory to Solr.  While I do update the index frequently, I also optimize frequently - its a little strange to me that this problem is showing up now after four weeks of zero problems.  Any suggestions/ideas would be very much appreciated!

Thanks,
Harish
Reply | Threaded
Open this post in threaded view
|

Re: Solr OpenBitSet OutofMemory Error

Otis Gospodnetic-2

Hi,

I'm not sure if anyone will be able to help without more detail.  First suggestion would be to look at Solr with a debugger/profiler to see where memory is used up.

Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch



----- Original Message ----

> From: smock <[hidden email]>
> To: [hidden email]
> Sent: Wednesday, March 25, 2009 2:37:26 PM
> Subject: Solr OpenBitSet OutofMemory Error
>
>
> Hello,
> After running a nightly release from around January of Solr for about 4
> weeks without any problems, I'm starting to see OutofMemory errors:
>
> Mar 24, 2009 1:35:36 AM org.apache.solr.common.SolrException log
> SEVERE: java.lang.OutOfMemoryError: Java heap space
>         at org.apache.lucene.util.OpenBitSet.clone(OpenBitSet.java:640)
>
>
> Is this a common error to see?  I'm running a lot of faceted queries on an
> index with about 7.5 million documents.  I'm giving about 8GBs of memory to
> Solr.  While I do update the index frequently, I also optimize frequently -
> its a little strange to me that this problem is showing up now after four
> weeks of zero problems.  Any suggestions/ideas would be very much
> appreciated!
>
> Thanks,
> Harish
> --
> View this message in context:
> http://www.nabble.com/Solr-OpenBitSet-OutofMemory-Error-tp22707576p22707576.html
> Sent from the Solr - User mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|

Re: Solr OpenBitSet OutofMemory Error

harish.agarwal
Hi Otis,

Thanks for the feedback - I'm pretty new to Java, could you or anyone else give me some pointers on how to run Solr with a debugger/profiler?  It would be really appreciated.

More generally, is OpenBitSet a utility for UnInvertedFields?  Is it reasonable to expect that this has something to do with faceted searches?

Thanks,
-Harish

Otis Gospodnetic wrote
Hi,

I'm not sure if anyone will be able to help without more detail.  First suggestion would be to look at Solr with a debugger/profiler to see where memory is used up.

Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch



----- Original Message ----
> From: smock <harish.agarwal@gmail.com>
> To: solr-user@lucene.apache.org
> Sent: Wednesday, March 25, 2009 2:37:26 PM
> Subject: Solr OpenBitSet OutofMemory Error
>
>
> Hello,
> After running a nightly release from around January of Solr for about 4
> weeks without any problems, I'm starting to see OutofMemory errors:
>
> Mar 24, 2009 1:35:36 AM org.apache.solr.common.SolrException log
> SEVERE: java.lang.OutOfMemoryError: Java heap space
>         at org.apache.lucene.util.OpenBitSet.clone(OpenBitSet.java:640)
>
>
> Is this a common error to see?  I'm running a lot of faceted queries on an
> index with about 7.5 million documents.  I'm giving about 8GBs of memory to
> Solr.  While I do update the index frequently, I also optimize frequently -
> its a little strange to me that this problem is showing up now after four
> weeks of zero problems.  Any suggestions/ideas would be very much
> appreciated!
>
> Thanks,
> Harish
> --
> View this message in context:
> http://www.nabble.com/Solr-OpenBitSet-OutofMemory-Error-tp22707576p22707576.html
> Sent from the Solr - User mailing list archive at Nabble.com.
Reply | Threaded
Open this post in threaded view
|

Re: Solr OpenBitSet OutofMemory Error

Otis Gospodnetic-2

Harish,

Yes, UnInvertedField uses OpenBitSet.
As for the profiler, YouKit is a good one - http://www.youkit.com/

Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch



----- Original Message ----

> From: smock <[hidden email]>
> To: [hidden email]
> Sent: Thursday, March 26, 2009 9:58:36 AM
> Subject: Re: Solr OpenBitSet OutofMemory Error
>
>
> Hi Otis,
>
> Thanks for the feedback - I'm pretty new to Java, could you or anyone else
> give me some pointers on how to run Solr with a debugger/profiler?  It would
> be really appreciated.
>
> More generally, is OpenBitSet a utility for UnInvertedFields?  Is it
> reasonable to expect that this has something to do with faceted searches?
>
> Thanks,
> -Harish
>
>
> Otis Gospodnetic wrote:
> >
> >
> > Hi,
> >
> > I'm not sure if anyone will be able to help without more detail.  First
> > suggestion would be to look at Solr with a debugger/profiler to see where
> > memory is used up.
> >
> > Otis
> > --
> > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
> >
> >
> >
> > ----- Original Message ----
> >> From: smock
> >> To: [hidden email]
> >> Sent: Wednesday, March 25, 2009 2:37:26 PM
> >> Subject: Solr OpenBitSet OutofMemory Error
> >>
> >>
> >> Hello,
> >> After running a nightly release from around January of Solr for about 4
> >> weeks without any problems, I'm starting to see OutofMemory errors:
> >>
> >> Mar 24, 2009 1:35:36 AM org.apache.solr.common.SolrException log
> >> SEVERE: java.lang.OutOfMemoryError: Java heap space
> >>         at org.apache.lucene.util.OpenBitSet.clone(OpenBitSet.java:640)
> >>
> >>
> >> Is this a common error to see?  I'm running a lot of faceted queries on
> >> an
> >> index with about 7.5 million documents.  I'm giving about 8GBs of memory
> >> to
> >> Solr.  While I do update the index frequently, I also optimize frequently
> >> -
> >> its a little strange to me that this problem is showing up now after four
> >> weeks of zero problems.  Any suggestions/ideas would be very much
> >> appreciated!
> >>
> >> Thanks,
> >> Harish
> >> --
> >> View this message in context:
> >>
> http://www.nabble.com/Solr-OpenBitSet-OutofMemory-Error-tp22707576p22707576.html
> >> Sent from the Solr - User mailing list archive at Nabble.com.
> >
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/Solr-OpenBitSet-OutofMemory-Error-tp22707576p22722631.html
> Sent from the Solr - User mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|

Re: Solr OpenBitSet OutofMemory Error

Yonik Seeley-2-2
On Thu, Mar 26, 2009 at 10:58 AM, Otis Gospodnetic
<[hidden email]> wrote:
> Yes, UnInvertedField uses OpenBitSet.

Right, for those terms that match a large percent of the documents.
But filtering (fq params) also takes up space, so you don't want the
filterCache too large.
Look at the stats page in solr admin... what are the entries for
fieldValueCache after you have done faceting on all fields you
normally facet on?
For example:
item_cat : {field=cat,memSize=5648,tindexSize=52,time=86,phase1=86,nTerms=19,bigTerms=10,termInstances=8,uses=1}

The "bigTerms" entry is the number of terms that were big enough to
use the fieldCache for.

-Yonik
http://www.lucidimagination.com