Hitting solr throttling for ingestion

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Hitting solr throttling for ingestion

shushuai zhu-2
Hi Experts,
We are using solr 8.4 (none cloud). When ingesting data with multiple processes to one core in a solr node, we are hitting some throttling: the max ingestion rate achieved is about 47K docs per second with 17 posting processes; each doc is about 250 bytes; the CPU utilization rate is only 20% and I/O about 6%. When increasing the posting processes, the posting will start failing. With solr 6.6, such issue does not happen: increasing posting processes will increase CPU/IO utilization rates to be close to 100% then start failing. 
Below are some relevant configurations specified in solrconfig.xml:
<indexConfig>  <maxIndexingThreads>16</maxIndexingThreads>  <ramBufferSizeMB>1024</ramBufferSizeMB>  <mergePolicyFactory class="org.apache.solr.index.TieredMergePolicyFactory">    <int name="maxMergeAtOnce">10</int>    <int name="segmentsPerTier">10</int>    <double name="maxMergedSegmentMB">4096</double>    <double name="noCFSRatio">0.1</double>    <double name="maxCFSSegmentSizeMB">4096</double>  </mergePolicyFactory>  <lockType>${solr.lock.type:native}</lockType>  <infoStream>true</infoStream></indexConfig>
<updateHandler class="solr.DirectUpdateHandler2">  <updateLog>    <str name="dir">${solr.ulog.dir:}</str>    <int name="numVersionBuckets">${solr.ulog.numVersionBuckets:65536}</int>  </updateLog>
  <autoCommit>    <maxTime>${solr.autoCommit.maxTime:120000}</maxTime>    <openSearcher>false</openSearcher>  </autoCommit>
  <autoSoftCommit>    <maxTime>${solr.autoSoftCommit.maxTime:5000}</maxTime>  </autoSoftCommit></updateHandler>
It seems maxIndexingThreads is no longer supported in solr 8? Any idea to break the solr throttling? Thanks. 
Shushuai