Java heap space

classic Classic list List threaded Threaded
35 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Java heap space

Marcus Stratmann
Hello,

I'm currently testing a large index with more than 10 million
documents and 24 fields, using the example installation with
Jetty.
When deleting or updateing documents from the index or doing
search queries I get "Java heap space" error messages like
this (in this case while trying to delete documents):


HTTP/1.1 500 Java+heap+space
Content-Type: text/html
Content-Length: 1236
Connection: close
Server: Jetty(6.0.x)

<html>
<head>
<title>Error 500 Java heap space</title>
</head>
<body>
<h2>HTTP ERROR: 500</h2><pre>Java heap space</pre>
<p>RequestURI=/solr/update</p>
<p><i><small><a href="http://jetty.mortbay.org">Powered by Jetty://</a></small></i></p>
</body>
</html>


This seems to be an error message generated by Jetty itself,
while I sometimes get another error message, which seems to
be generated by Solr:


HTTP/1.1 200 OK
Content-Type: text/xml;charset=UTF-8
Connection: close
Server: Jetty(6.0.x)

<result status="1">java.lang.OutOfMemoryError: Java heap space
</result>


In both cases obviously the server ran out of heap space.
I'm wondering what I can do to prevent this. I started the
server using the java options "-server -Xmx1024m".

Does anybody else have problems with heap space using a large
index? Is there anything I can do against this?

Thanks in advance,
Marcus

Reply | Threaded
Open this post in threaded view
|

Re: Java heap space

Chris Hostetter-3
: I'm currently testing a large index with more than 10 million
: documents and 24 fields, using the example installation with
: Jetty.
: When deleting or updateing documents from the index or doing
: search queries I get "Java heap space" error messages like
: this (in this case while trying to delete documents):
        ...
: In both cases obviously the server ran out of heap space.
: I'm wondering what I can do to prevent this. I started the
: server using the java options "-server -Xmx1024m".
:
: Does anybody else have problems with heap space using a large
: index? Is there anything I can do against this?

How big is your physical index directory on disk?

Generally speaking, there isn't much you can do to improve upon the memory
needs of a Lucene index that Solr isn't allready doing: Reuse a single
IndexSearcher as much as possible.

Your best bet is to allocate as much ram to the server as you can.
Depending on how full your caches are, and what hitratios you are getting
(the "STATISTICS" link from the Admin screen will tell you) you might want
to make some of them smaller to reduce the amount of RAM Solr uses for
them.

From an acctual index standpoint, if you don't care about doc/field boosts
of lengthNorms, then the omitNorm="true" option on your fields (or
fieldtypes) will help save one byte per document per field you use it on.


-Hoss

Reply | Threaded
Open this post in threaded view
|

Re: Java heap space

Marcus Stratmann
Chris Hostetter wrote:
> How big is your physical index directory on disk?
It's about 2.9G now.
Is there a direct connection between size of index and usage of ram?

> Your best bet is to allocate as much ram to the server as you can.
> Depending on how full your caches are, and what hitratios you are getting
> (the "STATISTICS" link from the Admin screen will tell you) you might want
> to make some of them smaller to reduce the amount of RAM Solr uses for
> them.
Hm, after disabling all caches I still get OutOfMemoryErrors.
All I do currently while testing is to delete documents. No searching or
inserting. Typically after deleting about 20,000 documents the server
throws the first error message.

>>From an acctual index standpoint, if you don't care about doc/field boosts
> of lengthNorms, then the omitNorm="true" option on your fields (or
> fieldtypes) will help save one byte per document per field you use it on.
That is something I could test, though I think this won't significantly
change the size of the index.

One thing that appears suspicious to me is that everything went fine as
long as the number of documents was below 10 million. Problems started
when this limit was exceeded. But maybe this is just a coincidence.

Marcus
Reply | Threaded
Open this post in threaded view
|

Re: Java heap space

Chris Hostetter-3
: > How big is your physical index directory on disk?
: It's about 2.9G now.
: Is there a direct connection between size of index and usage of ram?

Generally yes.  Lucene loads a lot of your index into memory ... not
neccessarily the "stored" fields, but quite a bit of the index structure
needed forsearching ... not to mention things like the FieldCache if you
start sorting by specific fields.

You may want to consult the lucene user list to get more advice about the
minimum amount of RAM recommended just for using a lucene index of size
___G in a JVM ... personally, I've always made sure my JVM was
allocated a max Heap size at least twice as big as my acctual index ...
mainly a paranoia thing on my part.

: Hm, after disabling all caches I still get OutOfMemoryErrors.
: All I do currently while testing is to delete documents. No searching or
: inserting. Typically after deleting about 20,000 documents the server
: throws the first error message.

interesting .. are you getting the OutOfMemory on an actual delete
operation or when doing a commit after executing some deletes?

part of the problem may be that under the covers, any delete involves
doing a query (even if oyou are deleting by uniqueKey, that's implimented
as a delete by Term, which requires iterating over a TermEnum to find the
relevent document, and if your index is big enough, loading that TermEnum
and may be the cause of your OOM.

: One thing that appears suspicious to me is that everything went fine as
: long as the number of documents was below 10 million. Problems started
: when this limit was exceeded. But maybe this is just a coincidence.

Maybe, maybe not ... what options are you using in your solrconfig.xml's
indexDefaults and mainIndex blocks? ... 10 million documents could be the
magic point at which your mergeFactor triggers the merging of several
large segments into one uber segment -- which may be big enough to cause
an OOM when the IndexReader tries to open it.

(Doug, Yonik, and Erik understand the underlying lucene memory usange
better then i do, hopefully they'll chime in with some advice)


-Hoss

Reply | Threaded
Open this post in threaded view
|

Re: Java heap space

Marcus Stratmann
Chris Hostetter wrote:
> interesting .. are you getting the OutOfMemory on an actual delete
> operation or when doing a commit after executing some deletes?

Yes, on a delete operation. I'm not doing any commits until the end of
all delete operations.
After reading this I was curious if using commits during deleting would
have any effect. So I tested doing a commit after 10,000 deletes at a
time (which, I know, is not recommended). But that simply didn't change
anything.

Meanwhile I found out that I can gain 10,000 documents more to delete
(before getting an OOM) by increasing the heap space by 500M.
Unfortunately we need to delete about 200,000 documents on each update
which would need 10G to be added to the heap space. Not to speak of
the same number of inserts.


> part of the problem may be that under the covers, any delete involves
> doing a query (even if oyou are deleting by uniqueKey, that's implimented
> s a delete by Term, which requires iterating over a TermEnum to find the
> relevent document, and if your index is big enough, loading that TermEnum
> and may be the cause of your OOM.

Yes, I thought so, too. And in fact I get OOM even if I just submit search
queries.


> Maybe, maybe not ... what options are you using in your solrconfig.xml's
> indexDefaults and mainIndex blocks?

I adopted the default values from the example installation which looked
quite reasonable to me.


> ... 10 million documents could be the
> magic point at which your mergeFactor triggers the merging of several
> large segments into one uber segment -- which may be big enough to cause
> an OOM when the IndexReader tries to open it.

Yes, I'm using the default mergeFactor of 10 and as 10 million is 10^7
this is what appeared suspicious to me.
Is it right, that the mergeFactor connot be changed once the index has
been built?

Marcus


Reply | Threaded
Open this post in threaded view
|

Re: Java heap space

Yonik Seeley
On 4/29/06, Marcus Stratmann <[hidden email]> wrote:
> Yes, on a delete operation. I'm not doing any commits until the end of
> all delete operations.

I assume this is a delete-by-id and not a delete-by-query?  They work
very differently.

There is some state stored for each pending delete-by-id... there is
a HashMap<String,Integer> with an entry for each id that needs to be
deleted.  This state shouldn't be that large though.

If fact, delete-by-id does nothing with a Lucene index at all until <commit/>

> After reading this I was curious if using commits during deleting would
> have any effect. So I tested doing a commit after 10,000 deletes at a
> time (which, I know, is not recommended). But that simply didn't change
> anything.

Strange... that suggests it's not the state kept in the HaspMap.

> Meanwhile I found out that I can gain 10,000 documents more to delete
> (before getting an OOM) by increasing the heap space by 500M.

Something else is going on.

> Unfortunately we need to delete about 200,000 documents on each update
> which would need 10G to be added to the heap space. Not to speak of
> the same number of inserts.

If you are first deleting so you can re-add a newer version of the
document, you don't need too... overwriting older documents based on
the uniqueKeyField is something Solr does for you!

> Yes, I thought so, too. And in fact I get OOM even if I just submit search
> queries.

Is it possible to use a profiler to see where all the memory is going?
It sounds like you may have uncovered a memory leak somewhere.
Also what OS, what JVM, what appserver are you using?


-Yonik
Reply | Threaded
Open this post in threaded view
|

Re: Java heap space

Marcus Stratmann
Yonik Seeley wrote:
>> Yes, on a delete operation. I'm not doing any commits until the end of
>> all delete operations.
> I assume this is a delete-by-id and not a delete-by-query?  They work
> very differently.

Yes, all queries are delete-by-id.


> If you are first deleting so you can re-add a newer version of the
> document, you don't need too... overwriting older documents based on
> the uniqueKeyField is something Solr does for you!

Yes, I know. But the articles in our (sql-)database get new IDs when
they are changed so they need to be deleted an re-inserted into the index.


> Is it possible to use a profiler to see where all the memory is going?
> It sounds like you may have uncovered a memory leak somewhere.

I'm not that experienced concerning Java, but maybe if you give me some
advice I'm glad if I can help. So far I had a quick look at JMP once but
that's all.
Don't hesitate to write me a PM on that subject.


> Also what OS, what JVM, what appserver are you using?
OS: Linux (Debian GNU/Linux i686)
JVM: Java HotSpot(TM) Server VM (build 1.5.0_06-b05, mixed mode) of
Sun's JDK 5.0.
Currently I'm using the Jetty installation from the solr nightly builds
for test purposes.

Marcus
Reply | Threaded
Open this post in threaded view
|

Re: Java heap space

Chris Hostetter-3

: > If you are first deleting so you can re-add a newer version of the
: > document, you don't need too... overwriting older documents based on
: > the uniqueKeyField is something Solr does for you!
:
: Yes, I know. But the articles in our (sql-)database get new IDs when
: they are changed so they need to be deleted an re-inserted into the index.

this is off the subject of the heap space issue ... but if the id changes,
then maybe it shouldn't be the uniqueId of your index? .. your code must
have someone of recognizing that article B with id 222 is a changed
version of article A with id 111 (otherwise how would you know to delete
111 when you insert 222?) ..whatever that mechanism is, perhaps it should
determine your uniqueKey?


-Hoss

Reply | Threaded
Open this post in threaded view
|

Re: Java heap space

Marcus Stratmann
Chris Hostetter wrote:
> this is off the subject of the heap space issue ... but if the id changes,
> then maybe it shouldn't be the uniqueId of your index? .. your code must
> have someone of recognizing that article B with id 222 is a changed
> version of article A with id 111 (otherwise how would you know to delete
> 111 when you insert 222?) ..whatever that mechanism is, perhaps it should
> determine your uniqueKey?

No, there is no "key" or something that reveals a relation between new
article B and old article A. After B is inserted and A is deleted, all
of A's existence is gone and we do not even know that B is A's
"successor". Changes are simply kept in a table which tells the system
which IDs to delete and which new (or changed) articles to insert,
automatically giving them new IDs. I know this may not be (or at least
sound) perfect and it is not the way things are handled normally. But
this works fine for our needs. We gather information about changes to
our data during the day and apply them on a nightly update (which, I
know, does not imply that IDs have to change).
So, yes, I'm sure I got the right uniqueKey. ;-)

Marcus



Reply | Threaded
Open this post in threaded view
|

Re: Java heap space

Marcus Stratmann
In reply to this post by Marcus Stratmann
Hello,

deleting or updating documents is still not possible for me so now I
tried to built a completely new index. Unfortunately this didn't work
either. Now I'm getting OOM after inserting slightly more than 20,000
documents to the new index.

To me this looks as if a bug has been introduced since the revision of
about 13th of april. To check this out, I looked for old builds but
there seem to be nightly-builds of the last four days only. Okay, so
next thing I tried was to get the code via svn. Unfortunately the code
does not compile ("package junit.framework does not exist"). I found out
that the last version I was able to complie was revision 393080
(2006-04-10). So I was neither able to get back the last (for me)
working revision nor to find out which revision this actually was.
Sorry, I would really like to help, but at the moment it seems Murphy is
striking.

Thanks,
Marcus
Reply | Threaded
Open this post in threaded view
|

Re: Java heap space

Yonik Seeley
Hi Marcus,

Is your problem reproducable with a test case you can share?
You could also try a different app-server like Tomcat to see if that
makes a difference.

What type is your id field defined to be?

-Yonik

On 5/3/06, Marcus Stratmann <[hidden email]> wrote:

> Hello,
>
> deleting or updating documents is still not possible for me so now I
> tried to built a completely new index. Unfortunately this didn't work
> either. Now I'm getting OOM after inserting slightly more than 20,000
> documents to the new index.
>
> To me this looks as if a bug has been introduced since the revision of
> about 13th of april. To check this out, I looked for old builds but
> there seem to be nightly-builds of the last four days only. Okay, so
> next thing I tried was to get the code via svn. Unfortunately the code
> does not compile ("package junit.framework does not exist"). I found out
> that the last version I was able to complie was revision 393080
> (2006-04-10). So I was neither able to get back the last (for me)
> working revision nor to find out which revision this actually was.
> Sorry, I would really like to help, but at the moment it seems Murphy is
> striking.
>
> Thanks,
> Marcus
Reply | Threaded
Open this post in threaded view
|

Re: Java heap space

Marcus Stratmann
Yonik Seeley wrote:
> Is your problem reproducable with a test case you can share?
Well, you can get the configuration files. If you ask for the data, this
could be a problem, since this is "real" data from our production
database. The amount of data needed could be another problem.

> You could also try a different app-server like Tomcat to see if that
> makes a difference.
This is another part of the problem because currently Tomcat won't work
with solr in my environment (a debian linux installation).

> What type is your id field defined to be?
<field name="buchID" type="slong" indexed="true" stored="true"/>
with slong defined by
<fieldtype name="slong" class="solr.SortableLongField"
sortMissingLast="true"/>
as in the sample schema.xml.

Marcus

Reply | Threaded
Open this post in threaded view
|

Re: Java heap space

Chris Hostetter-3
In reply to this post by Marcus Stratmann

: next thing I tried was to get the code via svn. Unfortunately the code
: does not compile ("package junit.framework does not exist"). I found out

This is because building a full Solr distribution from scratch requires
that you have JUnit.  Bt it is not required to run Solr.

: > Is your problem reproducable with a test case you can share?
: Well, you can get the configuration files. If you ask for the data, this
: could be a problem, since this is "real" data from our production
: database. The amount of data needed could be another problem.

What about generating random data dynamicly?

If you can submit a Jira issue, containing...

  1) Solr the config files you use.
  2) information on which ServletContainer you are testing this on
     (Tomcat, Jetty, etc..)
  3) information on how that servlet container is configured and run
     (ie: the container's config files ... or if you are just using the
     Jetty start.jar/configs that come in the Solr example say that)
  3) an app written in any common language (java, bash, perl, python,
     whatever) that generates and POSTs enough "random" documents to Solr
     to trigger your bug

...then other people can try to reproduce your error.

(Hopefully the bug you are encountering is specific to the configs, or the
Solr code, or the container -- and not a result of something strange in
your specific data, otherwise randomly generating fake data won't help).


-Hoss

Reply | Threaded
Open this post in threaded view
|

Re: Java heap space

Yonik Seeley
In reply to this post by Marcus Stratmann
I just tried sending in 100,000 deletes and it didn't cause a problem:
the memory grew from 22M to 30M.

Random thought: perhaps it has something to do with how you are
sending your requests?
If the client creates a new connection for each request, but doesn't
send the Connection:close header or close the connection after use,
these persistent connections can cause new threads to be created in
the app-server.

You can do a thread-dump with the latest nightly build to see if you
are accumulating too many threads in the appserver while the deletes
are going on (prob shouldn't be more than 20):
http://localhost:8983/solr/admin/threaddump.jsp

Here's the python script I used to try the 100,000 deletes... maybe
you can try it in your environment to see if it reproduces your memory
problem or not.  If it doesn't, look to your client code for possible
bugs.

-Yonik

--------- python script ----------
import httplib

class SolrConnection:
  def __init__(self, host='localhost:8983', solrBase='/solr'):
    self.host = host
    self.solrBase = solrBase
    #a socket is not really opened at this point.
    self.conn = httplib.HTTPConnection(self.host)
    self.conn.set_debuglevel(1000000)

  def doUpdateXML(self, request):
    self.conn.request('POST', self.solrBase+'/update', request)
    rsp = self.conn.getresponse()
    print rsp.status, rsp.reason
    data = rsp.read()
    print "data=",data

  def delete(self, id):
    xstr = '<delete><id>'+id+'</id></delete>'
    self.doUpdateXML(xstr)

c = SolrConnection()
for i in range(100000):
  c.delete(str(i))
Reply | Threaded
Open this post in threaded view
|

Re: Java heap space

Marcus Stratmann
In reply to this post by Chris Hostetter-3
Chris Hostetter wrote:
> This is because building a full Solr distribution from scratch requires
> that you have JUnit.  Bt it is not required to run Solr.
Ah, I see. That was a very valuable hint for me.
I was able now to compile an older revision (393957). Testing this
revision I was able to delete more than 600,000 documents without problems.
 From my point of view it looks like this: Revision 393957 works while
the latest revision cause problems. I don't know what part of the
distribution causes the problems but I will try to find out. I think a
good start would be to find out which was the first revision not working
for me. Maybe this would be enough information for you to find out what
had been changed at this point and what causes the problems.
I will also try just to change the solr.war to check if maybe Jetty is
responsible for the OOM.

I'll post a report when I have some results.

Marcus
Reply | Threaded
Open this post in threaded view
|

Re: Java heap space

Yonik Seeley
In reply to this post by Yonik Seeley
On 5/3/06, Yonik Seeley <[hidden email]> wrote:
> I just tried sending in 100,000 deletes and it didn't cause a problem:
> the memory grew from 22M to 30M.
>
> Random thought: perhaps it has something to do with how you are
> sending your requests?

Yep, I was able to reproduce a memory problem w/ Jetty on Linux when
using non-persistent connections (closed after each request).  The
same 100,000 deletes blew up the JVM to 1GB heap.

So this looks like it could be a Jetty problem (shame on me for using a beta).
I'm still not quite sure what changed in Solr that could make it
appear in later version and not in earlier versions though... the
version of Jetty is the same.

-Yonik
Reply | Threaded
Open this post in threaded view
|

Re: Java heap space

Yonik Seeley
I verified that Tomcat 5.5.17 doesn't experience this problem.

-Yonik

On 5/4/06, Yonik Seeley <[hidden email]> wrote:

> On 5/3/06, Yonik Seeley <[hidden email]> wrote:
> > I just tried sending in 100,000 deletes and it didn't cause a problem:
> > the memory grew from 22M to 30M.
> >
> > Random thought: perhaps it has something to do with how you are
> > sending your requests?
>
> Yep, I was able to reproduce a memory problem w/ Jetty on Linux when
> using non-persistent connections (closed after each request).  The
> same 100,000 deletes blew up the JVM to 1GB heap.
>
> So this looks like it could be a Jetty problem (shame on me for using a beta).
> I'm still not quite sure what changed in Solr that could make it
> appear in later version and not in earlier versions though... the
> version of Jetty is the same.
Reply | Threaded
Open this post in threaded view
|

Re: Java heap space

Bill Au
There seems to be a fair number of folks using the jetty with the example
app
as oppose to using Solr with their own appserver.  So I think it is best to
use a stable version of Jetty instead of the beta.  If no one objects, I can
go ahead and take care of this.

Bill

On 5/4/06, Yonik Seeley <[hidden email]> wrote:

>
> I verified that Tomcat 5.5.17 doesn't experience this problem.
>
> -Yonik
>
> On 5/4/06, Yonik Seeley <[hidden email]> wrote:
> > On 5/3/06, Yonik Seeley <[hidden email]> wrote:
> > > I just tried sending in 100,000 deletes and it didn't cause a problem:
> > > the memory grew from 22M to 30M.
> > >
> > > Random thought: perhaps it has something to do with how you are
> > > sending your requests?
> >
> > Yep, I was able to reproduce a memory problem w/ Jetty on Linux when
> > using non-persistent connections (closed after each request).  The
> > same 100,000 deletes blew up the JVM to 1GB heap.
> >
> > So this looks like it could be a Jetty problem (shame on me for using a
> beta).
> > I'm still not quite sure what changed in Solr that could make it
> > appear in later version and not in earlier versions though... the
> > version of Jetty is the same.
>
Reply | Threaded
Open this post in threaded view
|

Re: Java heap space

Erik Hatcher
Along these lines, locally I've been using the latest stable version  
of Jetty and it has worked fine, but I did see an "out of memory"  
exception the other day but have not seen it since so I'm not sure  
what caused it.

Moving to Tomcat, as long as we can configure it to be as lightweight  
as possible, is quite fine to me as well.

        Erik


On May 5, 2006, at 12:12 PM, Bill Au wrote:

> There seems to be a fair number of folks using the jetty with the  
> example
> app
> as oppose to using Solr with their own appserver.  So I think it is  
> best to
> use a stable version of Jetty instead of the beta.  If no one  
> objects, I can
> go ahead and take care of this.
>
> Bill
>
> On 5/4/06, Yonik Seeley <[hidden email]> wrote:
>>
>> I verified that Tomcat 5.5.17 doesn't experience this problem.
>>
>> -Yonik
>>
>> On 5/4/06, Yonik Seeley <[hidden email]> wrote:
>> > On 5/3/06, Yonik Seeley <[hidden email]> wrote:
>> > > I just tried sending in 100,000 deletes and it didn't cause a  
>> problem:
>> > > the memory grew from 22M to 30M.
>> > >
>> > > Random thought: perhaps it has something to do with how you are
>> > > sending your requests?
>> >
>> > Yep, I was able to reproduce a memory problem w/ Jetty on Linux  
>> when
>> > using non-persistent connections (closed after each request).  The
>> > same 100,000 deletes blew up the JVM to 1GB heap.
>> >
>> > So this looks like it could be a Jetty problem (shame on me for  
>> using a
>> beta).
>> > I'm still not quite sure what changed in Solr that could make it
>> > appear in later version and not in earlier versions though... the
>> > version of Jetty is the same.
>>

Reply | Threaded
Open this post in threaded view
|

Re: Java heap space

Bill Au
I was able to produce an OutOfMemoryError using Yonik's python script with
Jetty 6.
I was not able to do so with Jetty 5.1.11RC0, the latest stable version.  So
that's the
version of Jetty with which I will downgrade the Solr example app to.

Bill

On 5/5/06, Erik Hatcher <[hidden email]> wrote:

>
> Along these lines, locally I've been using the latest stable version
> of Jetty and it has worked fine, but I did see an "out of memory"
> exception the other day but have not seen it since so I'm not sure
> what caused it.
>
> Moving to Tomcat, as long as we can configure it to be as lightweight
> as possible, is quite fine to me as well.
>
>         Erik
>
>
> On May 5, 2006, at 12:12 PM, Bill Au wrote:
>
> > There seems to be a fair number of folks using the jetty with the
> > example
> > app
> > as oppose to using Solr with their own appserver.  So I think it is
> > best to
> > use a stable version of Jetty instead of the beta.  If no one
> > objects, I can
> > go ahead and take care of this.
> >
> > Bill
> >
> > On 5/4/06, Yonik Seeley <[hidden email]> wrote:
> >>
> >> I verified that Tomcat 5.5.17 doesn't experience this problem.
> >>
> >> -Yonik
> >>
> >> On 5/4/06, Yonik Seeley <[hidden email]> wrote:
> >> > On 5/3/06, Yonik Seeley <[hidden email]> wrote:
> >> > > I just tried sending in 100,000 deletes and it didn't cause a
> >> problem:
> >> > > the memory grew from 22M to 30M.
> >> > >
> >> > > Random thought: perhaps it has something to do with how you are
> >> > > sending your requests?
> >> >
> >> > Yep, I was able to reproduce a memory problem w/ Jetty on Linux
> >> when
> >> > using non-persistent connections (closed after each request).  The
> >> > same 100,000 deletes blew up the JVM to 1GB heap.
> >> >
> >> > So this looks like it could be a Jetty problem (shame on me for
> >> using a
> >> beta).
> >> > I'm still not quite sure what changed in Solr that could make it
> >> > appear in later version and not in earlier versions though... the
> >> > version of Jetty is the same.
> >>
>
>
12