Solr 7.5 - Indexing Failing due to "IndexWriter is Closed"

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

Solr 7.5 - Indexing Failing due to "IndexWriter is Closed"

Aroop Ganguly
Hi Group

I am facing this issue again.The stack mentions Heap space issue.

Are the document sizes too big ?

Not sure what I should be doing here; As on the solr admin ui I do not see jvm being anywhere close to being full.
Any advise on this is greatly welcome.


Full Stack trace:

2019-04-01 22:13:54.833 ERROR (qtp484199463-773)
 o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Server error writing document id C9C280C4-B3B7-4BEE-9EA5-C4925F5092D9 to the index
        at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:240)
        at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:67)
        at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55)
        at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:970)
        at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1186)
        at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:653)
        at org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103)
        at org.apache.solr.handler.loader.JavabinLoader$1.update(JavabinLoader.java:98)
        at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:188)
        at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readIterator(JavaBinUpdateRequestCodec.java:144)
        at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:311)
        at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:256)
        at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readNamedList(JavaBinUpdateRequestCodec.java:130)
        at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:276)
        at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:256)
        at org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:178)
        at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.unmarshal(JavaBinUpdateRequestCodec.java:195)
        at org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:109)
        at org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:55)
        at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:97)
        at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:2541)
        at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:709)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:515)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:377)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:323)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1634)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:219)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.server.Server.handle(Server.java:531)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
        at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
        at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:679)
        at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:693)
        at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1606)
        at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1601)
        at org.apache.solr.update.DirectUpdateHandler2.updateDocOrDocValues(DirectUpdateHandler2.java:964)
        at org.apache.solr.update.DirectUpdateHandler2.doNormalUpdate(DirectUpdateHandler2.java:341)
        at org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:288)
        at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:235)
        ... 60 more
Caused by: java.lang.OutOfMemoryError: Java heap space
        at org.apache.lucene.index.FieldInfos$Builder.getOrAdd(FieldInfos.java:413)
        at org.apache.lucene.index.DefaultIndexingChain.getOrAddField(DefaultIndexingChain.java:650)
        at org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:428)
        at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:394)
        at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:251)
        at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:494)
        at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1609)
        at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1601)
        at org.apache.solr.update.DirectUpdateHandler2.updateDocOrDocValues(DirectUpdateHandler2.java:964)
        at org.apache.solr.update.DirectUpdateHandler2.doNormalUpdate(DirectUpdateHandler2.java:341)
        at org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:288)
        at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:235)
        at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:67)
        at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55)
        at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:970)
        at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1186)
        at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:653)
        at org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103)
        at org.apache.solr.handler.loader.JavabinLoader$1.update(JavabinLoader.java:98)
        at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:188)
        at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readIterator(JavaBinUpdateRequestCodec.java:144)
        at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:311)
        at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:256)
        at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readNamedList(JavaBinUpdateRequestCodec.java:130)
        at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:276)
        at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:256)
        at org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:178)
        at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.unmarshal(JavaBinUpdateRequestCodec.java:195)
        at org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:109)
        at org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:55)
        at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:97)
        at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
Reply | Threaded
Open this post in threaded view
|

Solr 7.5 - Indexing Failing due to "IndexWriter is Closed"

Aroop Ganguly-3



Hi Group

I am facing this issue again.The stack mentions Heap space issue.

Are the document sizes too big ?

Not sure what I should be doing here; As on the solr admin ui I do not see jvm being anywhere close to being full.
Any advise on this is greatly welcome.


Full Stack trace:

2019-04-01 22:13:54.833 ERROR (qtp484199463-773)
 o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Server error writing document id C9C280C4-B3B7-4BEE-9EA5-C4925F5092D9 to the index
        at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:240)
        at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:67)
        at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55)
        at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:970)
        at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1186)
        at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:653)
        at org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103)
        at org.apache.solr.handler.loader.JavabinLoader$1.update(JavabinLoader.java:98)
        at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:188)
        at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readIterator(JavaBinUpdateRequestCodec.java:144)
        at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:311)
        at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:256)
        at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readNamedList(JavaBinUpdateRequestCodec.java:130)
        at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:276)
        at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:256)
        at org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:178)
        at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.unmarshal(JavaBinUpdateRequestCodec.java:195)
        at org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:109)
        at org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:55)
        at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:97)
        at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:2541)
        at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:709)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:515)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:377)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:323)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1634)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:219)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.server.Server.handle(Server.java:531)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
        at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
        at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:679)
        at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:693)
        at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1606)
        at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1601)
        at org.apache.solr.update.DirectUpdateHandler2.updateDocOrDocValues(DirectUpdateHandler2.java:964)
        at org.apache.solr.update.DirectUpdateHandler2.doNormalUpdate(DirectUpdateHandler2.java:341)
        at org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:288)
        at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:235)
        ... 60 more
Caused by: java.lang.OutOfMemoryError: Java heap space
        at org.apache.lucene.index.FieldInfos$Builder.getOrAdd(FieldInfos.java:413)
        at org.apache.lucene.index.DefaultIndexingChain.getOrAddField(DefaultIndexingChain.java:650)
        at org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:428)
        at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:394)
        at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:251)
        at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:494)
        at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1609)
        at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1601)
        at org.apache.solr.update.DirectUpdateHandler2.updateDocOrDocValues(DirectUpdateHandler2.java:964)
        at org.apache.solr.update.DirectUpdateHandler2.doNormalUpdate(DirectUpdateHandler2.java:341)
        at org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:288)
        at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:235)
        at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:67)
        at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55)
        at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:970)
        at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1186)
        at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:653)
        at org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103)
        at org.apache.solr.handler.loader.JavabinLoader$1.update(JavabinLoader.java:98)
        at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:188)
        at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readIterator(JavaBinUpdateRequestCodec.java:144)
        at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:311)
        at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:256)
        at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readNamedList(JavaBinUpdateRequestCodec.java:130)
        at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:276)
        at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:256)
        at org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:178)
        at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.unmarshal(JavaBinUpdateRequestCodec.java:195)
        at org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:109)
        at org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:55)
        at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:97)
        at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)

Reply | Threaded
Open this post in threaded view
|

Re: Solr 7.5 - Indexing Failing due to "IndexWriter is Closed"

Ashwin Tandel
In reply to this post by Aroop Ganguly
Please un-subscribe me.

Thanks in Advance,
Ashwin

On Mon, Apr 1, 2019 at 5:54 PM Aroop Ganguly <[hidden email]> wrote:

>
> Hi Group
>
> I am facing this issue again.The stack mentions Heap space issue.
>
> Are the document sizes too big ?
>
> Not sure what I should be doing here; As on the solr admin ui I do not see jvm being anywhere close to being full.
> Any advise on this is greatly welcome.
>
>
> Full Stack trace:
>
> 2019-04-01 22:13:54.833 ERROR (qtp484199463-773)
>  o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Server error writing document id C9C280C4-B3B7-4BEE-9EA5-C4925F5092D9 to the index
>         at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:240)
>         at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:67)
>         at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55)
>         at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:970)
>         at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1186)
>         at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:653)
>         at org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103)
>         at org.apache.solr.handler.loader.JavabinLoader$1.update(JavabinLoader.java:98)
>         at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:188)
>         at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readIterator(JavaBinUpdateRequestCodec.java:144)
>         at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:311)
>         at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:256)
>         at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readNamedList(JavaBinUpdateRequestCodec.java:130)
>         at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:276)
>         at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:256)
>         at org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:178)
>         at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.unmarshal(JavaBinUpdateRequestCodec.java:195)
>         at org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:109)
>         at org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:55)
>         at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:97)
>         at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
>         at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
>         at org.apache.solr.core.SolrCore.execute(SolrCore.java:2541)
>         at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:709)
>         at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:515)
>         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:377)
>         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:323)
>         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1634)
>         at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
>         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
>         at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
>         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>         at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
>         at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
>         at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
>         at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317)
>         at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
>         at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
>         at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
>         at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
>         at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219)
>         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
>         at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:219)
>         at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
>         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>         at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
>         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>         at org.eclipse.jetty.server.Server.handle(Server.java:531)
>         at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)
>         at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
>         at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
>         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
>         at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
>         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
>         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
>         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
>         at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
>         at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
>         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
>         at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
>         at java.base/java.lang.Thread.run(Thread.java:834)
> Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
>         at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:679)
>         at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:693)
>         at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1606)
>         at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1601)
>         at org.apache.solr.update.DirectUpdateHandler2.updateDocOrDocValues(DirectUpdateHandler2.java:964)
>         at org.apache.solr.update.DirectUpdateHandler2.doNormalUpdate(DirectUpdateHandler2.java:341)
>         at org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:288)
>         at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:235)
>         ... 60 more
> Caused by: java.lang.OutOfMemoryError: Java heap space
>         at org.apache.lucene.index.FieldInfos$Builder.getOrAdd(FieldInfos.java:413)
>         at org.apache.lucene.index.DefaultIndexingChain.getOrAddField(DefaultIndexingChain.java:650)
>         at org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:428)
>         at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:394)
>         at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:251)
>         at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:494)
>         at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1609)
>         at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1601)
>         at org.apache.solr.update.DirectUpdateHandler2.updateDocOrDocValues(DirectUpdateHandler2.java:964)
>         at org.apache.solr.update.DirectUpdateHandler2.doNormalUpdate(DirectUpdateHandler2.java:341)
>         at org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:288)
>         at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:235)
>         at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:67)
>         at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55)
>         at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:970)
>         at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1186)
>         at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:653)
>         at org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103)
>         at org.apache.solr.handler.loader.JavabinLoader$1.update(JavabinLoader.java:98)
>         at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:188)
>         at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readIterator(JavaBinUpdateRequestCodec.java:144)
>         at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:311)
>         at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:256)
>         at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readNamedList(JavaBinUpdateRequestCodec.java:130)
>         at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:276)
>         at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:256)
>         at org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:178)
>         at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.unmarshal(JavaBinUpdateRequestCodec.java:195)
>         at org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:109)
>         at org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:55)
>         at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:97)
>         at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
Reply | Threaded
Open this post in threaded view
|

Re: Solr 7.5 - Indexing Failing due to "IndexWriter is Closed"

Shawn Heisey-2
In reply to this post by Aroop Ganguly
On 4/1/2019 4:44 PM, Aroop Ganguly wrote:
> I am facing this issue again.The stack mentions Heap space issue.
>
> Are the document sizes too big ?
>
> Not sure what I should be doing here; As on the solr admin ui I do not see jvm being anywhere close to being full.
> Any advise on this is greatly welcome.

<snip>

> Caused by: java.lang.OutOfMemoryError: Java heap space

Java ran out of heap space.  This means that for what that process is
being asked to do, its heap is too small.  Solr needs more memory than
it is allowed to use.

There are exactly two things you can do.

1) Increase the heap size.
2) Change something so that less heap is required.

The second option is not always possible.

https://wiki.apache.org/solr/SolrPerformanceProblems#Java_Heap

Program operation is completely unpredictable when OOME strikes.  This
is why Solr is configured to self-destruct on OutOfMemoryError when it
is running on a non-Windows operating system.  We'd like the same thing
to happen for Windows, but don't have that capability yet.

Thanks,
Shawn
Reply | Threaded
Open this post in threaded view
|

Re: Solr 7.5 - Indexing Failing due to "IndexWriter is Closed"

Erick Erickson
In reply to this post by Ashwin Tandel
Please follow the instructions here: http://lucene.apache.org/solr/community.html#mailing-lists-irc
. You must use the _exact_ same e-mail as you used to subscribe.

If the initial try doesn't work and following the suggestions at the "problems" link doesn't work for you, let us know. But note you need to show us the _entire_ return header to allow anyone to diagnose the problem.

Best,
Erick

> On Apr 1, 2019, at 3:55 PM, Ashwin Tandel <[hidden email]> wrote:
>
> Please un-subscribe me.
>
> Thanks in Advance,
> Ashwin
>
> On Mon, Apr 1, 2019 at 5:54 PM Aroop Ganguly <[hidden email]> wrote:
>>
>> Hi Group
>>
>> I am facing this issue again.The stack mentions Heap space issue.
>>
>> Are the document sizes too big ?
>>
>> Not sure what I should be doing here; As on the solr admin ui I do not see jvm being anywhere close to being full.
>> Any advise on this is greatly welcome.
>>
>>
>> Full Stack trace:
>>
>> 2019-04-01 22:13:54.833 ERROR (qtp484199463-773)
>> o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Server error writing document id C9C280C4-B3B7-4BEE-9EA5-C4925F5092D9 to the index
>>        at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:240)
>>        at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:67)
>>        at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55)
>>        at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:970)
>>        at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1186)
>>        at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:653)
>>        at org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103)
>>        at org.apache.solr.handler.loader.JavabinLoader$1.update(JavabinLoader.java:98)
>>        at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:188)
>>        at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readIterator(JavaBinUpdateRequestCodec.java:144)
>>        at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:311)
>>        at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:256)
>>        at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readNamedList(JavaBinUpdateRequestCodec.java:130)
>>        at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:276)
>>        at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:256)
>>        at org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:178)
>>        at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.unmarshal(JavaBinUpdateRequestCodec.java:195)
>>        at org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:109)
>>        at org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:55)
>>        at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:97)
>>        at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
>>        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
>>        at org.apache.solr.core.SolrCore.execute(SolrCore.java:2541)
>>        at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:709)
>>        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:515)
>>        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:377)
>>        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:323)
>>        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1634)
>>        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
>>        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
>>        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
>>        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>>        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
>>        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
>>        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
>>        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317)
>>        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
>>        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
>>        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
>>        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
>>        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219)
>>        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
>>        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:219)
>>        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
>>        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>>        at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
>>        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>>        at org.eclipse.jetty.server.Server.handle(Server.java:531)
>>        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)
>>        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
>>        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
>>        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
>>        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
>>        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
>>        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
>>        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
>>        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
>>        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
>>        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
>>        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
>>        at java.base/java.lang.Thread.run(Thread.java:834)
>> Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
>>        at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:679)
>>        at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:693)
>>        at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1606)
>>        at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1601)
>>        at org.apache.solr.update.DirectUpdateHandler2.updateDocOrDocValues(DirectUpdateHandler2.java:964)
>>        at org.apache.solr.update.DirectUpdateHandler2.doNormalUpdate(DirectUpdateHandler2.java:341)
>>        at org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:288)
>>        at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:235)
>>        ... 60 more
>> Caused by: java.lang.OutOfMemoryError: Java heap space
>>        at org.apache.lucene.index.FieldInfos$Builder.getOrAdd(FieldInfos.java:413)
>>        at org.apache.lucene.index.DefaultIndexingChain.getOrAddField(DefaultIndexingChain.java:650)
>>        at org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:428)
>>        at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:394)
>>        at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:251)
>>        at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:494)
>>        at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1609)
>>        at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1601)
>>        at org.apache.solr.update.DirectUpdateHandler2.updateDocOrDocValues(DirectUpdateHandler2.java:964)
>>        at org.apache.solr.update.DirectUpdateHandler2.doNormalUpdate(DirectUpdateHandler2.java:341)
>>        at org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:288)
>>        at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:235)
>>        at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:67)
>>        at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55)
>>        at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:970)
>>        at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1186)
>>        at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:653)
>>        at org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103)
>>        at org.apache.solr.handler.loader.JavabinLoader$1.update(JavabinLoader.java:98)
>>        at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:188)
>>        at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readIterator(JavaBinUpdateRequestCodec.java:144)
>>        at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:311)
>>        at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:256)
>>        at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readNamedList(JavaBinUpdateRequestCodec.java:130)
>>        at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:276)
>>        at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:256)
>>        at org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:178)
>>        at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.unmarshal(JavaBinUpdateRequestCodec.java:195)
>>        at org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:109)
>>        at org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:55)
>>        at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:97)
>>        at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)

Reply | Threaded
Open this post in threaded view
|

Re: Solr 7.5 - Indexing Failing due to "IndexWriter is Closed"

Aroop Ganguly
In reply to this post by Shawn Heisey-2
Thanks Shawn, for the initial response.
Digging into a bit, I was wondering if we’d care to read the inner most stack.

From the inner most stack it seems to be telling us something about what trigger it ?
Ofcourse, the system could have been overloaded as well, but is the exception telling us something or its of no use to consider this stack


Caused by: java.lang.OutOfMemoryError: Java heap space
        at org.apache.lucene.index.FieldInfos$Builder.getOrAdd(FieldInfos.java:413)
        at org.apache.lucene.index.DefaultIndexingChain.getOrAddField(DefaultIndexingChain.java:650)
        at org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:428)
        at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:394)
        at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:251)
        at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:494)
        at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1609)
        at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1601)
        at org.apache.solr.update.DirectUpdateHandler2.updateDocOrDocValues(DirectUpdateHandler2.java:964)
        at org.apache.solr.update.DirectUpdateHandler2.doNormalUpdate(DirectUpdateHandler2.java:341)
        at org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:288)
        at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:235)
        at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:67)
        at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55)
        at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:970)
        at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1186)
        at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:653)
        at org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103)
        at org.apache.solr.handler.loader.JavabinLoader$1.update(JavabinLoader.java:98)
        at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:188)
        at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readIterator(JavaBinUpdateRequestCodec.java:144)
        at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:311)
        at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:256)
        at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readNamedList(JavaBinUpdateRequestCodec.java:130)
        at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:276)
        at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:256)
        at org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:178)
        at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.unmarshal(JavaBinUpdateRequestCodec.java:195)
        at org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:109)
        at org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:55)
        at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:97)
        at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)



> On Apr 1, 2019, at 4:06 PM, Shawn Heisey <[hidden email]> wrote:
>
> On 4/1/2019 4:44 PM, Aroop Ganguly wrote:
>> I am facing this issue again.The stack mentions Heap space issue.
>> Are the document sizes too big ?
>> Not sure what I should be doing here; As on the solr admin ui I do not see jvm being anywhere close to being full.
>> Any advise on this is greatly welcome.
>
> <snip>
>
>> Caused by: java.lang.OutOfMemoryError: Java heap space
>
> Java ran out of heap space.  This means that for what that process is being asked to do, its heap is too small.  Solr needs more memory than it is allowed to use.
>
> There are exactly two things you can do.
>
> 1) Increase the heap size.
> 2) Change something so that less heap is required.
>
> The second option is not always possible.
>
> https://wiki.apache.org/solr/SolrPerformanceProblems#Java_Heap
>
> Program operation is completely unpredictable when OOME strikes.  This is why Solr is configured to self-destruct on OutOfMemoryError when it is running on a non-Windows operating system.  We'd like the same thing to happen for Windows, but don't have that capability yet.
>
> Thanks,
> Shawn

Reply | Threaded
Open this post in threaded view
|

Re: Solr 7.5 - Indexing Failing due to "IndexWriter is Closed"

Shawn Heisey-2
4/1/2019 5:40 PM, Aroop Ganguly wrote:
> Thanks Shawn, for the initial response.
> Digging into a bit, I was wondering if we’d care to read the inner most stack.
>
>  From the inner most stack it seems to be telling us something about what trigger it ?
> Ofcourse, the system could have been overloaded as well, but is the exception telling us something or its of no use to consider this stack

The stacktrace on OOME is rarely useful.  The memory allocation where
the error is thrown probably has absolutely no connection to the part of
the program where major amounts of memory are being used.  It could be
ANY memory allocation that actually causes the error.

Thanks,
Shawn
Reply | Threaded
Open this post in threaded view
|

Re: Solr 7.5 - Indexing Failing due to "IndexWriter is Closed"

Aroop Ganguly
Turns out the cause was multiple indexing jobs indexing into the index simultaneously, which one can imagine can cause jvm loads on certain replicas for sure.
Once this was found and only one job ran at a time, things were back to normal.

Your comments seem right on no correlation to the stack trace!

> On Apr 1, 2019, at 5:32 PM, Shawn Heisey <[hidden email]> wrote:
>
> 4/1/2019 5:40 PM, Aroop Ganguly wrote:
>> Thanks Shawn, for the initial response.
>> Digging into a bit, I was wondering if we’d care to read the inner most stack.
>> From the inner most stack it seems to be telling us something about what trigger it ?
>> Ofcourse, the system could have been overloaded as well, but is the exception telling us something or its of no use to consider this stack
>
> The stacktrace on OOME is rarely useful.  The memory allocation where the error is thrown probably has absolutely no connection to the part of the program where major amounts of memory are being used.  It could be ANY memory allocation that actually causes the error.
>
> Thanks,
> Shawn

Reply | Threaded
Open this post in threaded view
|

Re: Solr 7.5 - Indexing Failing due to "IndexWriter is Closed"

Walter Underwood
We run multiple threads indexing to Solr all the time and have been doing so for years.

How big are your documents and how big are your batches?

wunder
Walter Underwood
[hidden email]
http://observer.wunderwood.org/  (my blog)

> On Apr 1, 2019, at 10:51 PM, Aroop Ganguly <[hidden email]> wrote:
>
> Turns out the cause was multiple indexing jobs indexing into the index simultaneously, which one can imagine can cause jvm loads on certain replicas for sure.
> Once this was found and only one job ran at a time, things were back to normal.
>
> Your comments seem right on no correlation to the stack trace!
>
>> On Apr 1, 2019, at 5:32 PM, Shawn Heisey <[hidden email]> wrote:
>>
>> 4/1/2019 5:40 PM, Aroop Ganguly wrote:
>>> Thanks Shawn, for the initial response.
>>> Digging into a bit, I was wondering if we’d care to read the inner most stack.
>>> From the inner most stack it seems to be telling us something about what trigger it ?
>>> Ofcourse, the system could have been overloaded as well, but is the exception telling us something or its of no use to consider this stack
>>
>> The stacktrace on OOME is rarely useful.  The memory allocation where the error is thrown probably has absolutely no connection to the part of the program where major amounts of memory are being used.  It could be ANY memory allocation that actually causes the error.
>>
>> Thanks,
>> Shawn
>

Reply | Threaded
Open this post in threaded view
|

Re: Solr 7.5 - Indexing Failing due to "IndexWriter is Closed"

Aroop Ganguly
Mutliple threads to the same index ? And how many concurrent threads?

Our case is not merely multiple threads but actually large scale spark indexer jobs that index 1B records at a time with a concurrency of 400.
In this case multiple such jobs were indexing into the same index.


> On Apr 2, 2019, at 7:25 AM, Walter Underwood <[hidden email]> wrote:
>
> We run multiple threads indexing to Solr all the time and have been doing so for years.
>
> How big are your documents and how big are your batches?
>
> wunder
> Walter Underwood
> [hidden email]
> http://observer.wunderwood.org/  (my blog)
>
>> On Apr 1, 2019, at 10:51 PM, Aroop Ganguly <[hidden email]> wrote:
>>
>> Turns out the cause was multiple indexing jobs indexing into the index simultaneously, which one can imagine can cause jvm loads on certain replicas for sure.
>> Once this was found and only one job ran at a time, things were back to normal.
>>
>> Your comments seem right on no correlation to the stack trace!
>>
>>> On Apr 1, 2019, at 5:32 PM, Shawn Heisey <[hidden email]> wrote:
>>>
>>> 4/1/2019 5:40 PM, Aroop Ganguly wrote:
>>>> Thanks Shawn, for the initial response.
>>>> Digging into a bit, I was wondering if we’d care to read the inner most stack.
>>>> From the inner most stack it seems to be telling us something about what trigger it ?
>>>> Ofcourse, the system could have been overloaded as well, but is the exception telling us something or its of no use to consider this stack
>>>
>>> The stacktrace on OOME is rarely useful.  The memory allocation where the error is thrown probably has absolutely no connection to the part of the program where major amounts of memory are being used.  It could be ANY memory allocation that actually causes the error.
>>>
>>> Thanks,
>>> Shawn
>>
>

Reply | Threaded
Open this post in threaded view
|

Re: Solr 7.5 - Indexing Failing due to "IndexWriter is Closed"

Walter Underwood
Yeah, that would overload it. To get good indexing speed, I configure two clients per CPU on the indexing machine. With one shard on a 16 processor machine, that would be 32 threads. With four shards on four 16 processor machines, 128 clients. Basically, one thread is waiting while the CPU processes a batch and the other is sending the next batch.

That should get the cluster to about 80% CPU. If the cluster is handling queries at the same time, I cut that way back, like one client thread for every two CPUs.

wunder
Walter Underwood
[hidden email]
http://observer.wunderwood.org/  (my blog)

> On Apr 2, 2019, at 8:13 PM, Aroop Ganguly <[hidden email]> wrote:
>
> Mutliple threads to the same index ? And how many concurrent threads?
>
> Our case is not merely multiple threads but actually large scale spark indexer jobs that index 1B records at a time with a concurrency of 400.
> In this case multiple such jobs were indexing into the same index.
>
>
>> On Apr 2, 2019, at 7:25 AM, Walter Underwood <[hidden email]> wrote:
>>
>> We run multiple threads indexing to Solr all the time and have been doing so for years.
>>
>> How big are your documents and how big are your batches?
>>
>> wunder
>> Walter Underwood
>> [hidden email]
>> http://observer.wunderwood.org/  (my blog)
>>
>>> On Apr 1, 2019, at 10:51 PM, Aroop Ganguly <[hidden email]> wrote:
>>>
>>> Turns out the cause was multiple indexing jobs indexing into the index simultaneously, which one can imagine can cause jvm loads on certain replicas for sure.
>>> Once this was found and only one job ran at a time, things were back to normal.
>>>
>>> Your comments seem right on no correlation to the stack trace!
>>>
>>>> On Apr 1, 2019, at 5:32 PM, Shawn Heisey <[hidden email]> wrote:
>>>>
>>>> 4/1/2019 5:40 PM, Aroop Ganguly wrote:
>>>>> Thanks Shawn, for the initial response.
>>>>> Digging into a bit, I was wondering if we’d care to read the inner most stack.
>>>>> From the inner most stack it seems to be telling us something about what trigger it ?
>>>>> Ofcourse, the system could have been overloaded as well, but is the exception telling us something or its of no use to consider this stack
>>>>
>>>> The stacktrace on OOME is rarely useful.  The memory allocation where the error is thrown probably has absolutely no connection to the part of the program where major amounts of memory are being used.  It could be ANY memory allocation that actually causes the error.
>>>>
>>>> Thanks,
>>>> Shawn
>>>
>>
>

Reply | Threaded
Open this post in threaded view
|

Re: Solr 7.5 - Indexing Failing due to "IndexWriter is Closed"

Aroop Ganguly
Thats an interesting scaling scheme you mention.
I have been trying to devise a good scheme for myself for our scale.

I will try to see how this works out for us.

> On Apr 2, 2019, at 9:15 PM, Walter Underwood <[hidden email]> wrote:
>
> Yeah, that would overload it. To get good indexing speed, I configure two clients per CPU on the indexing machine. With one shard on a 16 processor machine, that would be 32 threads. With four shards on four 16 processor machines, 128 clients. Basically, one thread is waiting while the CPU processes a batch and the other is sending the next batch.
>
> That should get the cluster to about 80% CPU. If the cluster is handling queries at the same time, I cut that way back, like one client thread for every two CPUs.
>
> wunder
> Walter Underwood
> [hidden email]
> http://observer.wunderwood.org/  (my blog)
>
>> On Apr 2, 2019, at 8:13 PM, Aroop Ganguly <[hidden email]> wrote:
>>
>> Mutliple threads to the same index ? And how many concurrent threads?
>>
>> Our case is not merely multiple threads but actually large scale spark indexer jobs that index 1B records at a time with a concurrency of 400.
>> In this case multiple such jobs were indexing into the same index.
>>
>>
>>> On Apr 2, 2019, at 7:25 AM, Walter Underwood <[hidden email]> wrote:
>>>
>>> We run multiple threads indexing to Solr all the time and have been doing so for years.
>>>
>>> How big are your documents and how big are your batches?
>>>
>>> wunder
>>> Walter Underwood
>>> [hidden email]
>>> http://observer.wunderwood.org/  (my blog)
>>>
>>>> On Apr 1, 2019, at 10:51 PM, Aroop Ganguly <[hidden email]> wrote:
>>>>
>>>> Turns out the cause was multiple indexing jobs indexing into the index simultaneously, which one can imagine can cause jvm loads on certain replicas for sure.
>>>> Once this was found and only one job ran at a time, things were back to normal.
>>>>
>>>> Your comments seem right on no correlation to the stack trace!
>>>>
>>>>> On Apr 1, 2019, at 5:32 PM, Shawn Heisey <[hidden email]> wrote:
>>>>>
>>>>> 4/1/2019 5:40 PM, Aroop Ganguly wrote:
>>>>>> Thanks Shawn, for the initial response.
>>>>>> Digging into a bit, I was wondering if we’d care to read the inner most stack.
>>>>>> From the inner most stack it seems to be telling us something about what trigger it ?
>>>>>> Ofcourse, the system could have been overloaded as well, but is the exception telling us something or its of no use to consider this stack
>>>>>
>>>>> The stacktrace on OOME is rarely useful.  The memory allocation where the error is thrown probably has absolutely no connection to the part of the program where major amounts of memory are being used.  It could be ANY memory allocation that actually causes the error.
>>>>>
>>>>> Thanks,
>>>>> Shawn
>>>>
>>>
>>
>

Reply | Threaded
Open this post in threaded view
|

Re: Solr 7.5 - Indexing Failing due to "IndexWriter is Closed"

Walter Underwood
If you have fast disk and enough RAM, indexing is CPU limited. So adjust the indexing load until the CPU is busy but not overloaded.

wunder
Walter Underwood
[hidden email]
http://observer.wunderwood.org/  (my blog)

> On Apr 2, 2019, at 9:23 PM, Aroop Ganguly <[hidden email]> wrote:
>
> Thats an interesting scaling scheme you mention.
> I have been trying to devise a good scheme for myself for our scale.
>
> I will try to see how this works out for us.
>
>> On Apr 2, 2019, at 9:15 PM, Walter Underwood <[hidden email]> wrote:
>>
>> Yeah, that would overload it. To get good indexing speed, I configure two clients per CPU on the indexing machine. With one shard on a 16 processor machine, that would be 32 threads. With four shards on four 16 processor machines, 128 clients. Basically, one thread is waiting while the CPU processes a batch and the other is sending the next batch.
>>
>> That should get the cluster to about 80% CPU. If the cluster is handling queries at the same time, I cut that way back, like one client thread for every two CPUs.
>>
>> wunder
>> Walter Underwood
>> [hidden email]
>> http://observer.wunderwood.org/  (my blog)
>>
>>> On Apr 2, 2019, at 8:13 PM, Aroop Ganguly <[hidden email]> wrote:
>>>
>>> Mutliple threads to the same index ? And how many concurrent threads?
>>>
>>> Our case is not merely multiple threads but actually large scale spark indexer jobs that index 1B records at a time with a concurrency of 400.
>>> In this case multiple such jobs were indexing into the same index.
>>>
>>>
>>>> On Apr 2, 2019, at 7:25 AM, Walter Underwood <[hidden email]> wrote:
>>>>
>>>> We run multiple threads indexing to Solr all the time and have been doing so for years.
>>>>
>>>> How big are your documents and how big are your batches?
>>>>
>>>> wunder
>>>> Walter Underwood
>>>> [hidden email]
>>>> http://observer.wunderwood.org/  (my blog)
>>>>
>>>>> On Apr 1, 2019, at 10:51 PM, Aroop Ganguly <[hidden email]> wrote:
>>>>>
>>>>> Turns out the cause was multiple indexing jobs indexing into the index simultaneously, which one can imagine can cause jvm loads on certain replicas for sure.
>>>>> Once this was found and only one job ran at a time, things were back to normal.
>>>>>
>>>>> Your comments seem right on no correlation to the stack trace!
>>>>>
>>>>>> On Apr 1, 2019, at 5:32 PM, Shawn Heisey <[hidden email]> wrote:
>>>>>>
>>>>>> 4/1/2019 5:40 PM, Aroop Ganguly wrote:
>>>>>>> Thanks Shawn, for the initial response.
>>>>>>> Digging into a bit, I was wondering if we’d care to read the inner most stack.
>>>>>>> From the inner most stack it seems to be telling us something about what trigger it ?
>>>>>>> Ofcourse, the system could have been overloaded as well, but is the exception telling us something or its of no use to consider this stack
>>>>>>
>>>>>> The stacktrace on OOME is rarely useful.  The memory allocation where the error is thrown probably has absolutely no connection to the part of the program where major amounts of memory are being used.  It could be ANY memory allocation that actually causes the error.
>>>>>>
>>>>>> Thanks,
>>>>>> Shawn
>>>>>
>>>>
>>>
>>
>