Re: Migration: SOLR8-Java8 -> SOLR8-JAVA11 indexing issue.

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: Migration: SOLR8-Java8 -> SOLR8-JAVA11 indexing issue.

Shawn Heisey-2
On 10/24/2019 11:50 AM, Junagade, Anup wrote:
>    *   Attachment 1: OpenJDK 8 vs OpenJDK 8 key metrics
>    *   Attachment 2:  OpenJDK 8 vs OpenJDK 8 waiting QTP Threads
>    *   Attachment 3: OpenJDK 11 Thread dump

There are no attachments.  Apache mailing lists swallow almost all
attachments.  You will need to use a file sharing website to
successfully get files to us.

> Heap allocated: 32 GB

If you set your heap to 31GB, you'll actually have more memory available
to Java than a heap size of 32GB.  This is because at 32GB, longer
pointers are required.  Solr has a tendency to create a very large
number of small objects, so the pointer size increase ends up using a
lot of memory.

> <mergePolicyFactory class="org.apache.solr.index.TieredMergePolicyFactory">
>            <int name="maxMergeAtOnce">100</int>
>            <int name="segmentsPerTier">150</int>
> </mergePolicyFactory>

These numbers are huge.  Without setting maxMergeAtOnceExplicit, you're
not getting the full benefit of increasing these settings beyond the
defaults of 10.  Set maxMergeAtOnce and segmentsPerTier to the same
number and then use three times that number for maxMergeAtOnceExplicit.
The Explicit setting is not mentioned in Solr documentation.  Numbers as
big as you have chosen will result in Solr keeping a LOT of files open,
becasue the index will end up with a large number of segments.  The OS
will definitely need to have its "max open files" limit increased.

> GC and ZK Settings
>
> -DzkClientTimeout=300000
> -DzkHost= XXXX,YYYY,ZZZZ
> -XX:+PrintGCDetails
> -XX:+UseG1GC
> -XX:+UseStringDeduplication
> -XX:ConcGCThreads=8
> -XX:InitiatingHeapOccupancyPercent=70
> -XX:MaxGCPauseMillis=200
> -XX:ParallelGCThreads=32
> -XX:PermSize=512m
> -Xlog:gc*:file=/var/solr/logs/solr_gc.log:time,
> uptime:filecount=9,
> filesize=20M
> -Xms32g-Xmx32g
> -Xss256k-Xss256k
> -verbose:gc

It looks like you have used your own GC settings instead of those that
Solr comes with.  Your settings are missing one of the most important
parameters for good GC performance.  You should let Solr's start script
handle GC tuning and GC logging without interference.

Thanks,
Shawn
Reply | Threaded
Open this post in threaded view
|

Re: Migration: SOLR8-Java8 -> SOLR8-JAVA11 indexing issue.

anup.junagade
Thanks Shawn for checking.

As advised we will execute the indexing with the new settings as mentioned
and will update the results.

Here are the links to missing attachments:

Attachment 1:  OpenJDK 11 vs OpenJDK 8 key metrics
<https://drive.google.com/open?id=1YRlP-vBrxAJ7NpyY_DkjPemU6gigsYkt>  
Attachment 2:   OpenJDK 11 vs OpenJDK 8 waiting QTP Threads
<https://drive.google.com/open?id=10eYjlveQyuepTbwU6ZByIgGX4dRNCkDe>  
Attachment 3:  OpenJDK 11 Thread dump
<https://drive.google.com/open?id=1aLaWDR9vPMSqPavO1z8ZSzuLocUkp7L3>  



--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html