Solr JVM Turning - 7.2.1

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Solr JVM Turning - 7.2.1

Sethuraman, Ganesh
Hi,

We are using Solr 7.2.1 with 2 nodes (245GB RAM each) and 3 node ZK cluster in production. We are using Java 8 with default GC settings (with NewRatio=3) with 15GB max heap, changed to 16 GB after the performance issue mentioned below.

We have about 90 collections in this (~8  shards each), about 50 of them are actively being used. About 3 collections are being actively updated using SolrJ update query with soft commit of 30 secs. Other collection go through update handler batch CSV update.

We had read timeout/slowness issue when Young Collection size usage peaked. As you can see in the GC Graph below during the problem time. After that we increased the overall heap size to 16GB (from 15 GB) and as you can see that we did not see any read issue.

  1.  I see our Heap is very large, we are seeing higher usage of young collection, is this due to solrj updates (concurrent one record update)?
  2.  Should we change the NewRatio to 2 (so that young size increases more)? as we are seeing only 58% usage of old gen
  3.  We are also seeing a behavior that if we restart the Solr in production, when updates are happening, one server starts up, but does not have all collections and shards up, and when we restart both the server up, it comes up fine, is this behavior also related to the Solrj updates?



Problem GC Report  https://gceasy.io/my-gc-report.jsp?p=YXJjaGl2ZWQvMjAxOS8xMC83Ly0tMDJfc29scl9nYy5sb2cuNi5jdXJyZW50LS0xNC00My01OA==&channel=WEB<https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgceasy.io%2Fmy-gc-report.jsp%3Fp%3DYXJjaGl2ZWQvMjAxOS8xMC83Ly0tMDJfc29scl9nYy5sb2cuNi5jdXJyZW50LS0xNC00My01OA%3D%3D%26channel%3DWEB&data=02%7C01%7Csethuramang%40dnb.com%7Cc0086a8014ba4c980faa08d752440eaf%7C19e2b708bf12437597198dec42771b3e%7C0%7C0%7C637068324577120430&sdata=TOum%2Fqf8ast2OAnFDpqumRniODNIl1elfBGYA%2BSgzYE%3D&reserved=0>

No Problem GC Report (still see higher Young collection use)  https://gceasy.io/my-gc-report.jsp?p=YXJjaGl2ZWQvMjAxOS8xMC85Ly0tMDJfX3NvbHJfZ2MubG9nLjIuY3VycmVudC0tMjAtNDQtMjY=&channel=WEB<https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgceasy.io%2Fmy-gc-report.jsp%3Fp%3DYXJjaGl2ZWQvMjAxOS8xMC85Ly0tMDJfX3NvbHJfZ2MubG9nLjIuY3VycmVudC0tMjAtNDQtMjY%3D%26channel%3DWEB&data=02%7C01%7Csethuramang%40dnb.com%7Cc0086a8014ba4c980faa08d752440eaf%7C19e2b708bf12437597198dec42771b3e%7C0%7C0%7C637068324577120430&sdata=AOqgAgpgBWri3a77%2F9I6JdrGzNkUpLltUdiZA056zPc%3D&reserved=0>

 Any help on the above question appreciated.

Thanks &Regards,

Ganesh




Reply | Threaded
Open this post in threaded view
|

RE: Solr JVM Turning - 7.2.1

Sethuraman, Ganesh
Solr Users,

Any suggestion or insights on the Solr behavior will help.

Regards
Ganesh

-----Original Message-----
From: Sethuraman, Ganesh <[hidden email]>
Sent: Wednesday, October 16, 2019 9:25 PM
To: [hidden email]
Subject: Solr JVM Turning - 7.2.1

CAUTION: This email originated from outside of D&B. Please do not click links or open attachments unless you recognize the sender and know the content is safe.


Hi,

We are using Solr 7.2.1 with 2 nodes (245GB RAM each) and 3 node ZK cluster in production. We are using Java 8 with default GC settings (with NewRatio=3) with 15GB max heap, changed to 16 GB after the performance issue mentioned below.

We have about 90 collections in this (~8  shards each), about 50 of them are actively being used. About 3 collections are being actively updated using SolrJ update query with soft commit of 30 secs. Other collection go through update handler batch CSV update.

We had read timeout/slowness issue when Young Collection size usage peaked. As you can see in the GC Graph below during the problem time. After that we increased the overall heap size to 16GB (from 15 GB) and as you can see that we did not see any read issue.

  1.  I see our Heap is very large, we are seeing higher usage of young collection, is this due to solrj updates (concurrent one record update)?
  2.  Should we change the NewRatio to 2 (so that young size increases more)? as we are seeing only 58% usage of old gen
  3.  We are also seeing a behavior that if we restart the Solr in production, when updates are happening, one server starts up, but does not have all collections and shards up, and when we restart both the server up, it comes up fine, is this behavior also related to the Solrj updates?



Problem GC Report  https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgceasy.io%2Fmy-gc-report.jsp%3Fp%3DYXJjaGl2ZWQvMjAxOS8xMC83Ly0tMDJfc29scl9nYy5sb2cuNi5jdXJyZW50LS0xNC00My01OA%3D%3D%26channel%3DWEB&amp;data=02%7C01%7CSethuramanG%40dnb.com%7C0cd17ad89bdd4227909108d752a0d0ef%7C19e2b708bf12437597198dec42771b3e%7C0%7C1%7C637068722973512337&amp;sdata=TrrmhXqtgbv9%2BKbcNulKw%2FrOCzTf9%2FSxO3JStWRlNG8%3D&amp;reserved=0<https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgceasy.io%2Fmy-gc-report.jsp%3Fp%3DYXJjaGl2ZWQvMjAxOS8xMC83Ly0tMDJfc29scl9nYy5sb2cuNi5jdXJyZW50LS0xNC00My01OA%3D%3D%26channel%3DWEB&amp;data=02%7C01%7CSethuramanG%40dnb.com%7C0cd17ad89bdd4227909108d752a0d0ef%7C19e2b708bf12437597198dec42771b3e%7C0%7C1%7C637068722973512337&amp;sdata=TrrmhXqtgbv9%2BKbcNulKw%2FrOCzTf9%2FSxO3JStWRlNG8%3D&amp;reserved=0>

No Problem GC Report (still see higher Young collection use)  https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgceasy.io%2Fmy-gc-report.jsp%3Fp%3DYXJjaGl2ZWQvMjAxOS8xMC85Ly0tMDJfX3NvbHJfZ2MubG9nLjIuY3VycmVudC0tMjAtNDQtMjY%3D%26channel%3DWEB&amp;data=02%7C01%7CSethuramanG%40dnb.com%7C0cd17ad89bdd4227909108d752a0d0ef%7C19e2b708bf12437597198dec42771b3e%7C0%7C1%7C637068722973517327&amp;sdata=MolLF1OSc8SBsx9rQqSfaHgwtzd534q%2B1Zqoc4W44QY%3D&amp;reserved=0<https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgceasy.io%2Fmy-gc-report.jsp%3Fp%3DYXJjaGl2ZWQvMjAxOS8xMC85Ly0tMDJfX3NvbHJfZ2MubG9nLjIuY3VycmVudC0tMjAtNDQtMjY%3D%26channel%3DWEB&amp;data=02%7C01%7CSethuramanG%40dnb.com%7C0cd17ad89bdd4227909108d752a0d0ef%7C19e2b708bf12437597198dec42771b3e%7C0%7C1%7C637068722973517327&amp;sdata=MolLF1OSc8SBsx9rQqSfaHgwtzd534q%2B1Zqoc4W44QY%3D&amp;reserved=0>

 Any help on the above question appreciated.

Thanks &Regards,

Ganesh