native Thread - solr 8.2.0

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

native Thread - solr 8.2.0

Joe Obernberger
Getting this error on some of the nodes in a solr cloud during heavy
indexing:

null:org.apache.solr.common.SolrException: Server error writing document id COLLECT20005437492077_activemq:queue:PAXTwitterExtractionQueue to the index
        at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:241)
        at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:76)
        at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55)
        at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:257)
        at org.apache.solr.update.processor.DistributedUpdateProcessor.doVersionAdd(DistributedUpdateProcessor.java:487)
        at org.apache.solr.update.processor.DistributedUpdateProcessor.lambda$versionAdd$0(DistributedUpdateProcessor.java:337)
        at org.apache.solr.update.VersionBucket.runWithLock(VersionBucket.java:50)
        at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:337)
        at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:223)
        at org.apache.solr.update.processor.DistributedZkUpdateProcessor.processAdd(DistributedZkUpdateProcessor.java:231)
        at org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103)
        at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55)
        at org.apache.solr.update.processor.AddSchemaFieldsUpdateProcessorFactory$AddSchemaFieldsUpdateProcessor.processAdd(AddSchemaFieldsUpdateProcessorFactory.java:475)
        at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55)
        at org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:118)
        at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55)
        at org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:118)
        at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55)
        at org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:118)
        at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55)
        at org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:118)
        at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55)
        at org.apache.solr.update.processor.FieldNameMutatingUpdateProcessorFactory$1.processAdd(FieldNameMutatingUpdateProcessorFactory.java:75)
        at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55)
        at org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:118)
        at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55)
        at org.apache.solr.update.processor.AbstractDefaultValueUpdateProcessorFactory$DefaultValueUpdateProcessor.processAdd(AbstractDefaultValueUpdateProcessorFactory.java:92)
        at org.apache.solr.handler.loader.JavabinLoader$1.update(JavabinLoader.java:110)
        at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$StreamingCodec.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:327)
        at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$StreamingCodec.readIterator(JavaBinUpdateRequestCodec.java:280)
        at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:336)
        at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:281)
        at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$StreamingCodec.readNamedList(JavaBinUpdateRequestCodec.java:235)
        at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:301)
        at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:281)
        at org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:194)
        at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.unmarshal(JavaBinUpdateRequestCodec.java:126)
        at org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:123)
        at org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:70)
        at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:97)
        at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:2578)
        at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:780)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:566)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:423)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:350)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1711)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1347)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1678)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1249)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:152)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.server.Server.handle(Server.java:505)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:781)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:917)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
        at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:681)
        at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:695)
        at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1591)
        at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1586)
        at org.apache.solr.update.DirectUpdateHandler2.updateDocOrDocValues(DirectUpdateHandler2.java:964)
        at org.apache.solr.update.DirectUpdateHandler2.doNormalUpdate(DirectUpdateHandler2.java:342)
        at org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:289)
        at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:236)
        ... 80 more
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start0(Native Method)
        at java.lang.Thread.start(Thread.java:717)
        at org.apache.hadoop.hdfs.DFSOutputStream.start(DFSOutputStream.java:780)
        at org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:316)
        at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1176)
        at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1155)
        at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1093)
        at org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:463)
        at org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:460)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:474)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1150)
        at org.apache.solr.store.hdfs.HdfsFileWriter.getOutputStream(HdfsFileWriter.java:51)
        at org.apache.solr.store.hdfs.HdfsFileWriter.<init>(HdfsFileWriter.java:40)
        at org.apache.solr.store.hdfs.HdfsDirectory.createOutput(HdfsDirectory.java:114)
        at org.apache.lucene.store.FilterDirectory.createOutput(FilterDirectory.java:74)
        at org.apache.solr.store.blockcache.BlockDirectory.createOutput(BlockDirectory.java:351)
        at org.apache.lucene.store.NRTCachingDirectory.unCache(NRTCachingDirectory.java:301)
        at org.apache.lucene.store.NRTCachingDirectory.sync(NRTCachingDirectory.java:156)
        at org.apache.lucene.store.LockValidatingDirectoryWrapper.sync(LockValidatingDirectoryWrapper.java:68)
        at org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:4804)
        at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:3276)
        at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3444)
        at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3409)
        at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:671)
        at org.apache.solr.update.CommitTracker.run(CommitTracker.java:270)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        ... 1 more

Server is running with these parameters:

-DDSTOP.KEY=solrrocks
-DSTOP.PORT=8100
-Dhost=puck
-Djava.library.path=/opt/cloudera/parcels/CDH/lib/hadoop/lib/native
-Djetty.home=/opt/solr8/server
-Djetty.port=9100
-Dsolr.autoCommit.maxTime=1800000
-Dsolr.autoSoftCommit.maxTime=120000
-Dsolr.clustering.enabled=true
-Dsolr.data.home=
-Dsolr.default.confdir=/opt/solr8/server/solr/configsets/_default/conf
-Dsolr.install.dir=/opt/solr8
-Dsolr.jetty.https.port=9100
-Dsolr.lock.type=hdfs
-Dsolr.log.dir=/opt/solr8/server/logs
-Dsolr.log.muteconsole
-Dsolr.solr.home=/etc/solr8
-Dsolr.solr.home=/opt/solr8/server/solr
-Duser.timezone=UTC
-DzkClientTimeout=300000
-DzkHost=frodo.querymasters.com:2181,bilbo.querymasters.com:2181,gandalf.querymasters.com:2181,cordelia.querymasters.com:2181,cressida.querymasters.com:2181/solr8.2.0
-XX:+AggressiveOpts
-XX:+ParallelRefProcEnabled
-XX:+PerfDisableSharedMem
-XX:+PrintGCApplicationStoppedTime
-XX:+PrintGCDateStamps
-XX:+PrintGCDetails
-XX:+PrintGCTimeStamps
-XX:+PrintHeapAtGC
-XX:+PrintTenuringDistribution
-XX:+UnlockExperimentalVMOptions
-XX:+UseG1GC
-XX:+UseGCLogFileRotation
-XX:+UseLargePages
-XX:-ResizePLAB
-XX:G1HeapRegionSize=16m
-XX:GCLogFileSize=20M
-XX:InitiatingHeapOccupancyPercent=75
-XX:MaxDirectMemorySize=8g
-XX:MaxGCPauseMillis=500
-XX:NumberOfGCLogFiles=9
-XX:OnOutOfMemoryError=/opt/solr8/bin/oom_solr.sh 9100 /opt/solr8/server/logs
-XX:ParallelGCThreads=8-Xloggc:/opt/solr8/server/logs/solr_gc.log-Xms20g-Xmx25g-Xss256k
-verbose:gc

Any ideas?
Thanks.

-Joe

Reply | Threaded
Open this post in threaded view
|

Re: native Thread - solr 8.2.0

Shawn Heisey-2
On 12/9/2019 2:23 PM, Joe Obernberger wrote:
> Getting this error on some of the nodes in a solr cloud during heavy
> indexing:

<snip>

> Caused by: java.lang.OutOfMemoryError: unable to create new native thread

Java was not able to start a new thread.  Most likely this is caused by
the operating system imposing limits on the number of processes or
threads that a user is allowed to start.

On Linux, the default limit is usually 1024 processes.  It doesn't take
much for a Solr install to need more threads than that.

How to increase the limit will depend on what OS you're running on.
Typically on Linux, this is controlled by /etc/security/limits.conf.  If
you're not on Linux, then you'll need to research how to increase the
process limit.

As long as you're fiddling with limits, you'll probably also want to
increase the open file limit.

Thanks,
Shawn
Reply | Threaded
Open this post in threaded view
|

Re: native Thread - solr 8.2.0

Mikhail Khludnev-2
My experience with  "OutOfMemoryError: unable to create new native thread"
as follows: it occurs on envs where devs refuse to use threadpools in favor
of old good new Thread().
Then, it turns rather interesting: If there are plenty of heap, GC doesn't
sweep Thread instances. Since they are native in Java, every of them hold
some ram for native stack. That exceeds stack space at some point of time.
So, check how many thread JVM hold after this particular OOME occurs by
jstack; you can even force GC to release that native stack space. Then,
rewrite the app, or reduce heap to enforce  GC.

On Tue, Dec 10, 2019 at 9:44 AM Shawn Heisey <[hidden email]> wrote:

> On 12/9/2019 2:23 PM, Joe Obernberger wrote:
> > Getting this error on some of the nodes in a solr cloud during heavy
> > indexing:
>
> <snip>
>
> > Caused by: java.lang.OutOfMemoryError: unable to create new native thread
>
> Java was not able to start a new thread.  Most likely this is caused by
> the operating system imposing limits on the number of processes or
> threads that a user is allowed to start.
>
> On Linux, the default limit is usually 1024 processes.  It doesn't take
> much for a Solr install to need more threads than that.
>
> How to increase the limit will depend on what OS you're running on.
> Typically on Linux, this is controlled by /etc/security/limits.conf.  If
> you're not on Linux, then you'll need to research how to increase the
> process limit.
>
> As long as you're fiddling with limits, you'll probably also want to
> increase the open file limit.
>
> Thanks,
> Shawn
>


--
Sincerely yours
Mikhail Khludnev
Reply | Threaded
Open this post in threaded view
|

Re: native Thread - solr 8.2.0

Walter Underwood
We’ve run into this fatal problem with 6.6 in prod. It gets overloaded, make 4000 threads, runs out of memory, and dies.

Not an acceptable design. Excess load MUST be rejected, otherwise the system goes into a stable congested state.

I was working with John Nagle when he figured this out in the late 1980s.

https://www.researchgate.net/publication/224734039_On_Packet_Switches_with_Infinite_Storage

wunder
Walter Underwood
[hidden email]
http://observer.wunderwood.org/  (my blog)

> On Dec 9, 2019, at 11:14 PM, Mikhail Khludnev <[hidden email]> wrote:
>
> My experience with  "OutOfMemoryError: unable to create new native thread"
> as follows: it occurs on envs where devs refuse to use threadpools in favor
> of old good new Thread().
> Then, it turns rather interesting: If there are plenty of heap, GC doesn't
> sweep Thread instances. Since they are native in Java, every of them hold
> some ram for native stack. That exceeds stack space at some point of time.
> So, check how many thread JVM hold after this particular OOME occurs by
> jstack; you can even force GC to release that native stack space. Then,
> rewrite the app, or reduce heap to enforce  GC.
>
> On Tue, Dec 10, 2019 at 9:44 AM Shawn Heisey <[hidden email]> wrote:
>
>> On 12/9/2019 2:23 PM, Joe Obernberger wrote:
>>> Getting this error on some of the nodes in a solr cloud during heavy
>>> indexing:
>>
>> <snip>
>>
>>> Caused by: java.lang.OutOfMemoryError: unable to create new native thread
>>
>> Java was not able to start a new thread.  Most likely this is caused by
>> the operating system imposing limits on the number of processes or
>> threads that a user is allowed to start.
>>
>> On Linux, the default limit is usually 1024 processes.  It doesn't take
>> much for a Solr install to need more threads than that.
>>
>> How to increase the limit will depend on what OS you're running on.
>> Typically on Linux, this is controlled by /etc/security/limits.conf.  If
>> you're not on Linux, then you'll need to research how to increase the
>> process limit.
>>
>> As long as you're fiddling with limits, you'll probably also want to
>> increase the open file limit.
>>
>> Thanks,
>> Shawn
>>
>
>
> --
> Sincerely yours
> Mikhail Khludnev

Reply | Threaded
Open this post in threaded view
|

Re: native Thread - solr 8.2.0

Erick Erickson
One other red flag is you’re apparently running in “schemaless” mode, based on:

org.apache.solr.update.processor.AddSchemaFieldsUpdateProcessorFactory$AddSchemaFieldsUpdateProcessor.processAdd(AddSchemaFieldsUpdateProcessorFactory.java:475)

When running in schemaless mode, if Solr encounters a field in a doc that it hasn’t seen before it will add a new field to the schema. Which will update the schema and reload the collection. If this is happening in the middle of “heavy indexing”, it’s going to clog up the works.

Please turn this off. See the message when you create a collection or look at the ref guide for how. The expense is one reason, but the second reason is that you have no control at all about how many fields you have in your index. Solr will merrily create these for _any_ new field. If you’re _lucky_, Solr will guess right. If you’re not lucky, Solr will start refusing to index documents due to field incompatibilities. Say the first value for a field is “1”. Solr guesses it’s an int. The next doc has “1.0”. solr will fail the doc.

Next up. When Solr has thousands of fields it starts to bog down due to housekeeping complexity. Do you have any idea how many fields have actually been realized in your index? 5? 50? 100K? The admin UI>>core>>schema will give you an idea.

Of course if your input docs are very tightly controlled, this really won’t be a problem, but in that case you don’t need schemaless anyway.

Why am I belaboring this? Because this may be the root of your thread issue. As you keep throwing docs at Solr, it has to queue them up if it’s making schema changes until the schema is updated and re-distributed to all replicas….

Best,
Erick

> On Dec 10, 2019, at 2:25 AM, Walter Underwood <[hidden email]> wrote:
>
> We’ve run into this fatal problem with 6.6 in prod. It gets overloaded, make 4000 threads, runs out of memory, and dies.
>
> Not an acceptable design. Excess load MUST be rejected, otherwise the system goes into a stable congested state.
>
> I was working with John Nagle when he figured this out in the late 1980s.
>
> https://www.researchgate.net/publication/224734039_On_Packet_Switches_with_Infinite_Storage
>
> wunder
> Walter Underwood
> [hidden email]
> http://observer.wunderwood.org/  (my blog)
>
>> On Dec 9, 2019, at 11:14 PM, Mikhail Khludnev <[hidden email]> wrote:
>>
>> My experience with  "OutOfMemoryError: unable to create new native thread"
>> as follows: it occurs on envs where devs refuse to use threadpools in favor
>> of old good new Thread().
>> Then, it turns rather interesting: If there are plenty of heap, GC doesn't
>> sweep Thread instances. Since they are native in Java, every of them hold
>> some ram for native stack. That exceeds stack space at some point of time.
>> So, check how many thread JVM hold after this particular OOME occurs by
>> jstack; you can even force GC to release that native stack space. Then,
>> rewrite the app, or reduce heap to enforce  GC.
>>
>> On Tue, Dec 10, 2019 at 9:44 AM Shawn Heisey <[hidden email]> wrote:
>>
>>> On 12/9/2019 2:23 PM, Joe Obernberger wrote:
>>>> Getting this error on some of the nodes in a solr cloud during heavy
>>>> indexing:
>>>
>>> <snip>
>>>
>>>> Caused by: java.lang.OutOfMemoryError: unable to create new native thread
>>>
>>> Java was not able to start a new thread.  Most likely this is caused by
>>> the operating system imposing limits on the number of processes or
>>> threads that a user is allowed to start.
>>>
>>> On Linux, the default limit is usually 1024 processes.  It doesn't take
>>> much for a Solr install to need more threads than that.
>>>
>>> How to increase the limit will depend on what OS you're running on.
>>> Typically on Linux, this is controlled by /etc/security/limits.conf.  If
>>> you're not on Linux, then you'll need to research how to increase the
>>> process limit.
>>>
>>> As long as you're fiddling with limits, you'll probably also want to
>>> increase the open file limit.
>>>
>>> Thanks,
>>> Shawn
>>>
>>
>>
>> --
>> Sincerely yours
>> Mikhail Khludnev
>

Reply | Threaded
Open this post in threaded view
|

Re: native Thread - solr 8.2.0

Joe Obernberger
Thank you Erick - it was a mistake for this collection to be running in
schemaless mode; I will fix that, but right now the 'PROCESSOR_LOGS'
schema only has 10 fields.  Another managed schema in the system has
over 1,000.

Shawn - I did see a post about setting vm.max_map_count higher (it was
65,530) and I increased it to 262144.  For the solr user, we're using
102,400 for open files and for max user processes, we use 65,000.

-Joe

On 12/10/2019 7:46 AM, Erick Erickson wrote:

> One other red flag is you’re apparently running in “schemaless” mode, based on:
>
> org.apache.solr.update.processor.AddSchemaFieldsUpdateProcessorFactory$AddSchemaFieldsUpdateProcessor.processAdd(AddSchemaFieldsUpdateProcessorFactory.java:475)
>
> When running in schemaless mode, if Solr encounters a field in a doc that it hasn’t seen before it will add a new field to the schema. Which will update the schema and reload the collection. If this is happening in the middle of “heavy indexing”, it’s going to clog up the works.
>
> Please turn this off. See the message when you create a collection or look at the ref guide for how. The expense is one reason, but the second reason is that you have no control at all about how many fields you have in your index. Solr will merrily create these for _any_ new field. If you’re _lucky_, Solr will guess right. If you’re not lucky, Solr will start refusing to index documents due to field incompatibilities. Say the first value for a field is “1”. Solr guesses it’s an int. The next doc has “1.0”. solr will fail the doc.
>
> Next up. When Solr has thousands of fields it starts to bog down due to housekeeping complexity. Do you have any idea how many fields have actually been realized in your index? 5? 50? 100K? The admin UI>>core>>schema will give you an idea.
>
> Of course if your input docs are very tightly controlled, this really won’t be a problem, but in that case you don’t need schemaless anyway.
>
> Why am I belaboring this? Because this may be the root of your thread issue. As you keep throwing docs at Solr, it has to queue them up if it’s making schema changes until the schema is updated and re-distributed to all replicas….
>
> Best,
> Erick
>
>> On Dec 10, 2019, at 2:25 AM, Walter Underwood <[hidden email]> wrote:
>>
>> We’ve run into this fatal problem with 6.6 in prod. It gets overloaded, make 4000 threads, runs out of memory, and dies.
>>
>> Not an acceptable design. Excess load MUST be rejected, otherwise the system goes into a stable congested state.
>>
>> I was working with John Nagle when he figured this out in the late 1980s.
>>
>> https://www.researchgate.net/publication/224734039_On_Packet_Switches_with_Infinite_Storage
>>
>> wunder
>> Walter Underwood
>> [hidden email]
>> http://observer.wunderwood.org/  (my blog)
>>
>>> On Dec 9, 2019, at 11:14 PM, Mikhail Khludnev <[hidden email]> wrote:
>>>
>>> My experience with  "OutOfMemoryError: unable to create new native thread"
>>> as follows: it occurs on envs where devs refuse to use threadpools in favor
>>> of old good new Thread().
>>> Then, it turns rather interesting: If there are plenty of heap, GC doesn't
>>> sweep Thread instances. Since they are native in Java, every of them hold
>>> some ram for native stack. That exceeds stack space at some point of time.
>>> So, check how many thread JVM hold after this particular OOME occurs by
>>> jstack; you can even force GC to release that native stack space. Then,
>>> rewrite the app, or reduce heap to enforce  GC.
>>>
>>> On Tue, Dec 10, 2019 at 9:44 AM Shawn Heisey <[hidden email]> wrote:
>>>
>>>> On 12/9/2019 2:23 PM, Joe Obernberger wrote:
>>>>> Getting this error on some of the nodes in a solr cloud during heavy
>>>>> indexing:
>>>> <snip>
>>>>
>>>>> Caused by: java.lang.OutOfMemoryError: unable to create new native thread
>>>> Java was not able to start a new thread.  Most likely this is caused by
>>>> the operating system imposing limits on the number of processes or
>>>> threads that a user is allowed to start.
>>>>
>>>> On Linux, the default limit is usually 1024 processes.  It doesn't take
>>>> much for a Solr install to need more threads than that.
>>>>
>>>> How to increase the limit will depend on what OS you're running on.
>>>> Typically on Linux, this is controlled by /etc/security/limits.conf.  If
>>>> you're not on Linux, then you'll need to research how to increase the
>>>> process limit.
>>>>
>>>> As long as you're fiddling with limits, you'll probably also want to
>>>> increase the open file limit.
>>>>
>>>> Thanks,
>>>> Shawn
>>>>
>>>
>>> --
>>> Sincerely yours
>>> Mikhail Khludnev
>