cleanup after OutOfMemoryError

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

cleanup after OutOfMemoryError

Ryan McKinley
I have an application where I am calling DirectUpdateHandler2 directly with:

  update.addDoc(cmd);

This will sometimes hit:

java.lang.OutOfMemoryError: Java heap space
at org.apache.lucene.util.UnicodeUtil.UTF16toUTF8(UnicodeUtil.java:248)
at org.apache.lucene.store.DataOutput.writeString(DataOutput.java:234)
at
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.writeField(CompressingStoredFieldsWriter.java:273)
at
org.apache.lucene.index.StoredFieldsProcessor.finishDocument(StoredFieldsProcessor.java:126)
at
org.apache.lucene.index.TwoStoredFieldsConsumers.finishDocument(TwoStoredFieldsConsumers.java:65)
at
org.apache.lucene.index.DocFieldProcessor.finishDocument(DocFieldProcessor.java:264)
at
org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:283)
at
org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:432)
at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1513)
at
org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:212)
at voyager.index.zmq.IndexingRunner.apply(IndexingRunner.java:303)

and then a little while later:

auto commit error...:java.lang.IllegalStateException: this writer hit an
OutOfMemoryError; cannot commit
at
org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2726)
at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2897)
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2872)
at
org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:549)
at org.apache.solr.update.CommitTracker.run(CommitTracker.java:216)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)


Is there anythign I can/should do to cleanup after the OOME?  At a minimum
I do not want any new requests using the same IndexWriter.  Should I use:


  catch(OutOfMemoryError ex) {

   update.getCommitTracker().cancelPendingCommit();
     update.newIndexWriter(false);
     ...

or perhaps 'true' for rollback?

Thanks
Ryan
Reply | Threaded
Open this post in threaded view
|

Re: cleanup after OutOfMemoryError

Mark Miller-3
I don't know that there is any 'safe' thing you can do other than restart -
but if I were to try anything, I would use true for rollback.

- Mark


On Wed, Sep 4, 2013 at 9:44 AM, Ryan McKinley <[hidden email]> wrote:

> I have an application where I am calling DirectUpdateHandler2 directly
> with:
>
>   update.addDoc(cmd);
>
> This will sometimes hit:
>
> java.lang.OutOfMemoryError: Java heap space
> at org.apache.lucene.util.UnicodeUtil.UTF16toUTF8(UnicodeUtil.java:248)
> at org.apache.lucene.store.DataOutput.writeString(DataOutput.java:234)
> at
>
> org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.writeField(CompressingStoredFieldsWriter.java:273)
> at
>
> org.apache.lucene.index.StoredFieldsProcessor.finishDocument(StoredFieldsProcessor.java:126)
> at
>
> org.apache.lucene.index.TwoStoredFieldsConsumers.finishDocument(TwoStoredFieldsConsumers.java:65)
> at
>
> org.apache.lucene.index.DocFieldProcessor.finishDocument(DocFieldProcessor.java:264)
> at
>
> org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:283)
> at
>
> org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:432)
> at
> org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1513)
> at
>
> org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:212)
> at voyager.index.zmq.IndexingRunner.apply(IndexingRunner.java:303)
>
> and then a little while later:
>
> auto commit error...:java.lang.IllegalStateException: this writer hit an
> OutOfMemoryError; cannot commit
> at
>
> org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2726)
> at
> org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2897)
> at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2872)
> at
>
> org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:549)
> at org.apache.solr.update.CommitTracker.run(CommitTracker.java:216)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>
>
> Is there anythign I can/should do to cleanup after the OOME?  At a minimum
> I do not want any new requests using the same IndexWriter.  Should I use:
>
>
>   catch(OutOfMemoryError ex) {
>
>    update.getCommitTracker().cancelPendingCommit();
>      update.newIndexWriter(false);
>      ...
>
> or perhaps 'true' for rollback?
>
> Thanks
> Ryan
>



--
- Mark