Throughput does not increase in spite of low CPU usage

classic Classic list List threaded Threaded
21 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Throughput does not increase in spite of low CPU usage

Yasufumi Mizoguchi
Hi,

I am trying some tests to confirm if single Solr instance can perform over
1000 queries per second(!).

But now, although CPU usage is 40% or so and iowait is almost 0%,
throughput does not increase over 60 queries per second.

I think there are some bottlenecks around Kernel, JVM, or Solr settings.

The values we already checked and configured are followings.

* Kernel:
file descriptor
net.ipv4.tcp_max_syn_backlog
net.ipv4.tcp_syncookies
net.core.somaxconn
net.core.rmem_max
net.core.wmem_max
net.ipv4.tcp_rmem
net.ipv4.tcp_wmem

* JVM:
Heap [ -> 32GB]
G1GC settings

* Solr:
(Jetty) MaxThreads [ -> 20000]


And the other info is as follows.

CPU : 16 cores
RAM : 128 GB
Disk : SSD 500GB
NIC : 10Gbps(maybe)
OS : Ubuntu 14.04
JVM : OpenJDK 1.8.0u191
Solr : 6.2.1
Index size : about 60GB

Any insights will be appreciated.

Thanks and regards,
Yasufumi.
Reply | Threaded
Open this post in threaded view
|

Re: Throughput does not increase in spite of low CPU usage

Deepak Goel
Hello

Can you please share the JVM heap settings in detail?

Deepak

On Mon, 30 Sep 2019, 11:15 Yasufumi Mizoguchi, <[hidden email]>
wrote:

> Hi,
>
> I am trying some tests to confirm if single Solr instance can perform over
> 1000 queries per second(!).
>
> But now, although CPU usage is 40% or so and iowait is almost 0%,
> throughput does not increase over 60 queries per second.
>
> I think there are some bottlenecks around Kernel, JVM, or Solr settings.
>
> The values we already checked and configured are followings.
>
> * Kernel:
> file descriptor
> net.ipv4.tcp_max_syn_backlog
> net.ipv4.tcp_syncookies
> net.core.somaxconn
> net.core.rmem_max
> net.core.wmem_max
> net.ipv4.tcp_rmem
> net.ipv4.tcp_wmem
>
> * JVM:
> Heap [ -> 32GB]
> G1GC settings
>
> * Solr:
> (Jetty) MaxThreads [ -> 20000]
>
>
> And the other info is as follows.
>
> CPU : 16 cores
> RAM : 128 GB
> Disk : SSD 500GB
> NIC : 10Gbps(maybe)
> OS : Ubuntu 14.04
> JVM : OpenJDK 1.8.0u191
> Solr : 6.2.1
> Index size : about 60GB
>
> Any insights will be appreciated.
>
> Thanks and regards,
> Yasufumi.
>
Reply | Threaded
Open this post in threaded view
|

Re: Throughput does not increase in spite of low CPU usage

Yasufumi Mizoguchi
Hi, Deepak.
Thank you for replying me.

JVM settings from solr.in.sh file are as follows. (Sorry, I could not show
all due to our policy)

-verbose:gc
-XX:+PrintHeapAtGC
-XX:+PrintGCDetails
-XX:+PrintGCDateStamps
-XX:+PrintGCTimeStamps
-XX:+PrintTenuringDistribution
-XX:+PrintGCApplicationStoppedTime
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.port=18983
-XX:OnOutOfMemoryError=/home/solr/solr-6.2.1/bin/oom_solr.sh
-XX:NewSize=128m
-XX:MaxNewSize=128m
-XX:+UseG1GC
-XX:+PerfDisableSharedMem
-XX:+ParallelRefProcEnabled
-XX:G1HeapRegionSize=8m
-XX:MaxGCPauseMillis=250
-XX:InitiatingHeapOccupancyPercent=75
-XX:+UseLargePages
-XX:+AggressiveOpts
-Xmx32G
-Xms32G
-Xss256k


Thanks & Regards
Yasufumi.

2019年9月30日(月) 16:12 Deepak Goel <[hidden email]>:

> Hello
>
> Can you please share the JVM heap settings in detail?
>
> Deepak
>
> On Mon, 30 Sep 2019, 11:15 Yasufumi Mizoguchi, <[hidden email]>
> wrote:
>
> > Hi,
> >
> > I am trying some tests to confirm if single Solr instance can perform
> over
> > 1000 queries per second(!).
> >
> > But now, although CPU usage is 40% or so and iowait is almost 0%,
> > throughput does not increase over 60 queries per second.
> >
> > I think there are some bottlenecks around Kernel, JVM, or Solr settings.
> >
> > The values we already checked and configured are followings.
> >
> > * Kernel:
> > file descriptor
> > net.ipv4.tcp_max_syn_backlog
> > net.ipv4.tcp_syncookies
> > net.core.somaxconn
> > net.core.rmem_max
> > net.core.wmem_max
> > net.ipv4.tcp_rmem
> > net.ipv4.tcp_wmem
> >
> > * JVM:
> > Heap [ -> 32GB]
> > G1GC settings
> >
> > * Solr:
> > (Jetty) MaxThreads [ -> 20000]
> >
> >
> > And the other info is as follows.
> >
> > CPU : 16 cores
> > RAM : 128 GB
> > Disk : SSD 500GB
> > NIC : 10Gbps(maybe)
> > OS : Ubuntu 14.04
> > JVM : OpenJDK 1.8.0u191
> > Solr : 6.2.1
> > Index size : about 60GB
> >
> > Any insights will be appreciated.
> >
> > Thanks and regards,
> > Yasufumi.
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Throughput does not increase in spite of low CPU usage

Ere Maijala
Just a side note: -Xmx32G is really bad for performance as it forces
Java to use non-compressed pointers. You'll actually get better results
with -Xmx31G. For more information, see e.g.
https://blog.codecentric.de/en/2014/02/35gb-heap-less-32gb-java-jvm-memory-oddities/

Regards,
Ere

Yasufumi Mizoguchi kirjoitti 30.9.2019 klo 11.05:

> Hi, Deepak.
> Thank you for replying me.
>
> JVM settings from solr.in.sh file are as follows. (Sorry, I could not show
> all due to our policy)
>
> -verbose:gc
> -XX:+PrintHeapAtGC
> -XX:+PrintGCDetails
> -XX:+PrintGCDateStamps
> -XX:+PrintGCTimeStamps
> -XX:+PrintTenuringDistribution
> -XX:+PrintGCApplicationStoppedTime
> -Dcom.sun.management.jmxremote.ssl=false
> -Dcom.sun.management.jmxremote.authenticate=false
> -Dcom.sun.management.jmxremote.port=18983
> -XX:OnOutOfMemoryError=/home/solr/solr-6.2.1/bin/oom_solr.sh
> -XX:NewSize=128m
> -XX:MaxNewSize=128m
> -XX:+UseG1GC
> -XX:+PerfDisableSharedMem
> -XX:+ParallelRefProcEnabled
> -XX:G1HeapRegionSize=8m
> -XX:MaxGCPauseMillis=250
> -XX:InitiatingHeapOccupancyPercent=75
> -XX:+UseLargePages
> -XX:+AggressiveOpts
> -Xmx32G
> -Xms32G
> -Xss256k
>
>
> Thanks & Regards
> Yasufumi.
>
> 2019年9月30日(月) 16:12 Deepak Goel <[hidden email]>:
>
>> Hello
>>
>> Can you please share the JVM heap settings in detail?
>>
>> Deepak
>>
>> On Mon, 30 Sep 2019, 11:15 Yasufumi Mizoguchi, <[hidden email]>
>> wrote:
>>
>>> Hi,
>>>
>>> I am trying some tests to confirm if single Solr instance can perform
>> over
>>> 1000 queries per second(!).
>>>
>>> But now, although CPU usage is 40% or so and iowait is almost 0%,
>>> throughput does not increase over 60 queries per second.
>>>
>>> I think there are some bottlenecks around Kernel, JVM, or Solr settings.
>>>
>>> The values we already checked and configured are followings.
>>>
>>> * Kernel:
>>> file descriptor
>>> net.ipv4.tcp_max_syn_backlog
>>> net.ipv4.tcp_syncookies
>>> net.core.somaxconn
>>> net.core.rmem_max
>>> net.core.wmem_max
>>> net.ipv4.tcp_rmem
>>> net.ipv4.tcp_wmem
>>>
>>> * JVM:
>>> Heap [ -> 32GB]
>>> G1GC settings
>>>
>>> * Solr:
>>> (Jetty) MaxThreads [ -> 20000]
>>>
>>>
>>> And the other info is as follows.
>>>
>>> CPU : 16 cores
>>> RAM : 128 GB
>>> Disk : SSD 500GB
>>> NIC : 10Gbps(maybe)
>>> OS : Ubuntu 14.04
>>> JVM : OpenJDK 1.8.0u191
>>> Solr : 6.2.1
>>> Index size : about 60GB
>>>
>>> Any insights will be appreciated.
>>>
>>> Thanks and regards,
>>> Yasufumi.
>>>
>>
>

--
Ere Maijala
Kansalliskirjasto / The National Library of Finland
Reply | Threaded
Open this post in threaded view
|

Re: Throughput does not increase in spite of low CPU usage

Yasufumi Mizoguchi
Hi, Ere.

Thank you for valuable feedback.
I will try Xmx31G and Xms31G instead of current ones.

Thanks and Regards,
Yasufumi.

2019年9月30日(月) 17:19 Ere Maijala <[hidden email]>:

> Just a side note: -Xmx32G is really bad for performance as it forces
> Java to use non-compressed pointers. You'll actually get better results
> with -Xmx31G. For more information, see e.g.
>
> https://blog.codecentric.de/en/2014/02/35gb-heap-less-32gb-java-jvm-memory-oddities/
>
> Regards,
> Ere
>
> Yasufumi Mizoguchi kirjoitti 30.9.2019 klo 11.05:
> > Hi, Deepak.
> > Thank you for replying me.
> >
> > JVM settings from solr.in.sh file are as follows. (Sorry, I could not
> show
> > all due to our policy)
> >
> > -verbose:gc
> > -XX:+PrintHeapAtGC
> > -XX:+PrintGCDetails
> > -XX:+PrintGCDateStamps
> > -XX:+PrintGCTimeStamps
> > -XX:+PrintTenuringDistribution
> > -XX:+PrintGCApplicationStoppedTime
> > -Dcom.sun.management.jmxremote.ssl=false
> > -Dcom.sun.management.jmxremote.authenticate=false
> > -Dcom.sun.management.jmxremote.port=18983
> > -XX:OnOutOfMemoryError=/home/solr/solr-6.2.1/bin/oom_solr.sh
> > -XX:NewSize=128m
> > -XX:MaxNewSize=128m
> > -XX:+UseG1GC
> > -XX:+PerfDisableSharedMem
> > -XX:+ParallelRefProcEnabled
> > -XX:G1HeapRegionSize=8m
> > -XX:MaxGCPauseMillis=250
> > -XX:InitiatingHeapOccupancyPercent=75
> > -XX:+UseLargePages
> > -XX:+AggressiveOpts
> > -Xmx32G
> > -Xms32G
> > -Xss256k
> >
> >
> > Thanks & Regards
> > Yasufumi.
> >
> > 2019年9月30日(月) 16:12 Deepak Goel <[hidden email]>:
> >
> >> Hello
> >>
> >> Can you please share the JVM heap settings in detail?
> >>
> >> Deepak
> >>
> >> On Mon, 30 Sep 2019, 11:15 Yasufumi Mizoguchi, <[hidden email]>
> >> wrote:
> >>
> >>> Hi,
> >>>
> >>> I am trying some tests to confirm if single Solr instance can perform
> >> over
> >>> 1000 queries per second(!).
> >>>
> >>> But now, although CPU usage is 40% or so and iowait is almost 0%,
> >>> throughput does not increase over 60 queries per second.
> >>>
> >>> I think there are some bottlenecks around Kernel, JVM, or Solr
> settings.
> >>>
> >>> The values we already checked and configured are followings.
> >>>
> >>> * Kernel:
> >>> file descriptor
> >>> net.ipv4.tcp_max_syn_backlog
> >>> net.ipv4.tcp_syncookies
> >>> net.core.somaxconn
> >>> net.core.rmem_max
> >>> net.core.wmem_max
> >>> net.ipv4.tcp_rmem
> >>> net.ipv4.tcp_wmem
> >>>
> >>> * JVM:
> >>> Heap [ -> 32GB]
> >>> G1GC settings
> >>>
> >>> * Solr:
> >>> (Jetty) MaxThreads [ -> 20000]
> >>>
> >>>
> >>> And the other info is as follows.
> >>>
> >>> CPU : 16 cores
> >>> RAM : 128 GB
> >>> Disk : SSD 500GB
> >>> NIC : 10Gbps(maybe)
> >>> OS : Ubuntu 14.04
> >>> JVM : OpenJDK 1.8.0u191
> >>> Solr : 6.2.1
> >>> Index size : about 60GB
> >>>
> >>> Any insights will be appreciated.
> >>>
> >>> Thanks and regards,
> >>> Yasufumi.
> >>>
> >>
> >
>
> --
> Ere Maijala
> Kansalliskirjasto / The National Library of Finland
>
Reply | Threaded
Open this post in threaded view
|

Re: Throughput does not increase in spite of low CPU usage

Erick Erickson
The most basic question is how you are load-testing it? Assuming you have some kind of client firing queries at Solr, keep adding threads so Solr is handling more and more queries in parallel. If you start to see the response time at the client get longer _and_ the  QTime in Solr’s response stays about the same, then the queries are queueing up and you need to see about increasing the Jetty threads handling queries.

Second is whether you’re hitting GC pauses, look at the GC logs, especially for “stop the world” pauses. This is unlikely as you’re still getting 60 qps, but something to check.

Setting your heap to 31G is good advice, but it won’t dramatically increase the throughput I’d guess.

If your I/O isn’t very high, then your index is mostly memory-resident. A general bit of tuning advice is to _reduce_ the heap size, leaving OS memory for the index. See Uwe’s blog: https://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html. There’s a sweet spot between having too much heap and too little, and unfortunately you have to experiment to find out.

But given the numbers you’re talking about, you won’t be getting 1,000 QPS on a single box and you’ll have to scale out with replicas to hit that number. Getting all the QPS you can out of the box is important, of course. Do be careful to use enough different queries that you don’t get them from the queryResultCache. I had one client who was thrilled they were getting 3ms response times…. by firing the same query over and over and hitting the queryResultCache 99.9999% of the time ;).

Best,
Erick

> On Sep 30, 2019, at 4:28 AM, Yasufumi Mizoguchi <[hidden email]> wrote:
>
> Hi, Ere.
>
> Thank you for valuable feedback.
> I will try Xmx31G and Xms31G instead of current ones.
>
> Thanks and Regards,
> Yasufumi.
>
> 2019年9月30日(月) 17:19 Ere Maijala <[hidden email]>:
>
>> Just a side note: -Xmx32G is really bad for performance as it forces
>> Java to use non-compressed pointers. You'll actually get better results
>> with -Xmx31G. For more information, see e.g.
>>
>> https://blog.codecentric.de/en/2014/02/35gb-heap-less-32gb-java-jvm-memory-oddities/
>>
>> Regards,
>> Ere
>>
>> Yasufumi Mizoguchi kirjoitti 30.9.2019 klo 11.05:
>>> Hi, Deepak.
>>> Thank you for replying me.
>>>
>>> JVM settings from solr.in.sh file are as follows. (Sorry, I could not
>> show
>>> all due to our policy)
>>>
>>> -verbose:gc
>>> -XX:+PrintHeapAtGC
>>> -XX:+PrintGCDetails
>>> -XX:+PrintGCDateStamps
>>> -XX:+PrintGCTimeStamps
>>> -XX:+PrintTenuringDistribution
>>> -XX:+PrintGCApplicationStoppedTime
>>> -Dcom.sun.management.jmxremote.ssl=false
>>> -Dcom.sun.management.jmxremote.authenticate=false
>>> -Dcom.sun.management.jmxremote.port=18983
>>> -XX:OnOutOfMemoryError=/home/solr/solr-6.2.1/bin/oom_solr.sh
>>> -XX:NewSize=128m
>>> -XX:MaxNewSize=128m
>>> -XX:+UseG1GC
>>> -XX:+PerfDisableSharedMem
>>> -XX:+ParallelRefProcEnabled
>>> -XX:G1HeapRegionSize=8m
>>> -XX:MaxGCPauseMillis=250
>>> -XX:InitiatingHeapOccupancyPercent=75
>>> -XX:+UseLargePages
>>> -XX:+AggressiveOpts
>>> -Xmx32G
>>> -Xms32G
>>> -Xss256k
>>>
>>>
>>> Thanks & Regards
>>> Yasufumi.
>>>
>>> 2019年9月30日(月) 16:12 Deepak Goel <[hidden email]>:
>>>
>>>> Hello
>>>>
>>>> Can you please share the JVM heap settings in detail?
>>>>
>>>> Deepak
>>>>
>>>> On Mon, 30 Sep 2019, 11:15 Yasufumi Mizoguchi, <[hidden email]>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I am trying some tests to confirm if single Solr instance can perform
>>>> over
>>>>> 1000 queries per second(!).
>>>>>
>>>>> But now, although CPU usage is 40% or so and iowait is almost 0%,
>>>>> throughput does not increase over 60 queries per second.
>>>>>
>>>>> I think there are some bottlenecks around Kernel, JVM, or Solr
>> settings.
>>>>>
>>>>> The values we already checked and configured are followings.
>>>>>
>>>>> * Kernel:
>>>>> file descriptor
>>>>> net.ipv4.tcp_max_syn_backlog
>>>>> net.ipv4.tcp_syncookies
>>>>> net.core.somaxconn
>>>>> net.core.rmem_max
>>>>> net.core.wmem_max
>>>>> net.ipv4.tcp_rmem
>>>>> net.ipv4.tcp_wmem
>>>>>
>>>>> * JVM:
>>>>> Heap [ -> 32GB]
>>>>> G1GC settings
>>>>>
>>>>> * Solr:
>>>>> (Jetty) MaxThreads [ -> 20000]
>>>>>
>>>>>
>>>>> And the other info is as follows.
>>>>>
>>>>> CPU : 16 cores
>>>>> RAM : 128 GB
>>>>> Disk : SSD 500GB
>>>>> NIC : 10Gbps(maybe)
>>>>> OS : Ubuntu 14.04
>>>>> JVM : OpenJDK 1.8.0u191
>>>>> Solr : 6.2.1
>>>>> Index size : about 60GB
>>>>>
>>>>> Any insights will be appreciated.
>>>>>
>>>>> Thanks and regards,
>>>>> Yasufumi.
>>>>>
>>>>
>>>
>>
>> --
>> Ere Maijala
>> Kansalliskirjasto / The National Library of Finland
>>

Reply | Threaded
Open this post in threaded view
|

Re: Throughput does not increase in spite of low CPU usage

Shawn Heisey-2
In reply to this post by Yasufumi Mizoguchi
On 9/29/2019 11:44 PM, Yasufumi Mizoguchi wrote:
> I am trying some tests to confirm if single Solr instance can perform over
> 1000 queries per second(!).

In general, I would never expect a single instance to handle a large
number of queries per second unless the index is REALLY small -- dozens
or hundreds of very small documents.  A 60GB index definitely does not
qualify.

I don't think it will be possible to handle 1000 queries per second with
a single server even on a really small index, but I've never actually tried.

> But now, although CPU usage is 40% or so and iowait is almost 0%,
> throughput does not increase over 60 queries per second.

A query rate of 60 per second is pretty good with an index size of 60GB.
  The low iowait would tend to confirm that the index is well cached by
the OS.

If you need to handle 1000 queries per second, you need more copies of
your index on additional Solr servers, with something in the mix to
perform load balancing.

Some thoughts:

With your -XX:MaxNewSize=128m setting, you are likely causing garbage
collection to occur VERY frequently, which will slow things down.
Solr's default GC settings include -XX:NewRatio=3 so that the new
generation will be much larger than what you have set.  A program like
Solr that allocates a lot of memory will need a fairly large new generation.

I agree with the idea of setting the heap to 31GB.  Setting it to 31GB
will actually leave more memory available to Solr than setting it to
32GB, because of the decreased pointer sizes.

Definitely check what Erick mentioned.  If you're seeing what he
described, adjusting how threads work might get you more throughput.
But look into your new generation sizing first.

Thanks,
Shawn
Reply | Threaded
Open this post in threaded view
|

Re: Throughput does not increase in spite of low CPU usage

Walter Underwood
In reply to this post by Yasufumi Mizoguchi
31G is still a very large heap. We use 8G for all of our different clusters.

Do you have JVM monitoring? Look at the heap used after a major GC. Use that number, plus some extra, for the heap size.

wunder
Walter Underwood
[hidden email]
http://observer.wunderwood.org/  (my blog)

> On Sep 30, 2019, at 1:28 AM, Yasufumi Mizoguchi <[hidden email]> wrote:
>
> Hi, Ere.
>
> Thank you for valuable feedback.
> I will try Xmx31G and Xms31G instead of current ones.
>
> Thanks and Regards,
> Yasufumi.
>
> 2019年9月30日(月) 17:19 Ere Maijala <[hidden email]>:
>
>> Just a side note: -Xmx32G is really bad for performance as it forces
>> Java to use non-compressed pointers. You'll actually get better results
>> with -Xmx31G. For more information, see e.g.
>>
>> https://blog.codecentric.de/en/2014/02/35gb-heap-less-32gb-java-jvm-memory-oddities/
>>
>> Regards,
>> Ere
>>
>> Yasufumi Mizoguchi kirjoitti 30.9.2019 klo 11.05:
>>> Hi, Deepak.
>>> Thank you for replying me.
>>>
>>> JVM settings from solr.in.sh file are as follows. (Sorry, I could not
>> show
>>> all due to our policy)
>>>
>>> -verbose:gc
>>> -XX:+PrintHeapAtGC
>>> -XX:+PrintGCDetails
>>> -XX:+PrintGCDateStamps
>>> -XX:+PrintGCTimeStamps
>>> -XX:+PrintTenuringDistribution
>>> -XX:+PrintGCApplicationStoppedTime
>>> -Dcom.sun.management.jmxremote.ssl=false
>>> -Dcom.sun.management.jmxremote.authenticate=false
>>> -Dcom.sun.management.jmxremote.port=18983
>>> -XX:OnOutOfMemoryError=/home/solr/solr-6.2.1/bin/oom_solr.sh
>>> -XX:NewSize=128m
>>> -XX:MaxNewSize=128m
>>> -XX:+UseG1GC
>>> -XX:+PerfDisableSharedMem
>>> -XX:+ParallelRefProcEnabled
>>> -XX:G1HeapRegionSize=8m
>>> -XX:MaxGCPauseMillis=250
>>> -XX:InitiatingHeapOccupancyPercent=75
>>> -XX:+UseLargePages
>>> -XX:+AggressiveOpts
>>> -Xmx32G
>>> -Xms32G
>>> -Xss256k
>>>
>>>
>>> Thanks & Regards
>>> Yasufumi.
>>>
>>> 2019年9月30日(月) 16:12 Deepak Goel <[hidden email]>:
>>>
>>>> Hello
>>>>
>>>> Can you please share the JVM heap settings in detail?
>>>>
>>>> Deepak
>>>>
>>>> On Mon, 30 Sep 2019, 11:15 Yasufumi Mizoguchi, <[hidden email]>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I am trying some tests to confirm if single Solr instance can perform
>>>> over
>>>>> 1000 queries per second(!).
>>>>>
>>>>> But now, although CPU usage is 40% or so and iowait is almost 0%,
>>>>> throughput does not increase over 60 queries per second.
>>>>>
>>>>> I think there are some bottlenecks around Kernel, JVM, or Solr
>> settings.
>>>>>
>>>>> The values we already checked and configured are followings.
>>>>>
>>>>> * Kernel:
>>>>> file descriptor
>>>>> net.ipv4.tcp_max_syn_backlog
>>>>> net.ipv4.tcp_syncookies
>>>>> net.core.somaxconn
>>>>> net.core.rmem_max
>>>>> net.core.wmem_max
>>>>> net.ipv4.tcp_rmem
>>>>> net.ipv4.tcp_wmem
>>>>>
>>>>> * JVM:
>>>>> Heap [ -> 32GB]
>>>>> G1GC settings
>>>>>
>>>>> * Solr:
>>>>> (Jetty) MaxThreads [ -> 20000]
>>>>>
>>>>>
>>>>> And the other info is as follows.
>>>>>
>>>>> CPU : 16 cores
>>>>> RAM : 128 GB
>>>>> Disk : SSD 500GB
>>>>> NIC : 10Gbps(maybe)
>>>>> OS : Ubuntu 14.04
>>>>> JVM : OpenJDK 1.8.0u191
>>>>> Solr : 6.2.1
>>>>> Index size : about 60GB
>>>>>
>>>>> Any insights will be appreciated.
>>>>>
>>>>> Thanks and regards,
>>>>> Yasufumi.
>>>>>
>>>>
>>>
>>
>> --
>> Ere Maijala
>> Kansalliskirjasto / The National Library of Finland
>>

Reply | Threaded
Open this post in threaded view
|

Re: Throughput does not increase in spite of low CPU usage

Deepak Goel
In reply to this post by Yasufumi Mizoguchi
Hello

Can you please try increasing 'new size' and 'max new size' to 1GB+?

Deepak

On Mon, 30 Sep 2019, 13:35 Yasufumi Mizoguchi, <[hidden email]>
wrote:

> Hi, Deepak.
> Thank you for replying me.
>
> JVM settings from solr.in.sh file are as follows. (Sorry, I could not show
> all due to our policy)
>
> -verbose:gc
> -XX:+PrintHeapAtGC
> -XX:+PrintGCDetails
> -XX:+PrintGCDateStamps
> -XX:+PrintGCTimeStamps
> -XX:+PrintTenuringDistribution
> -XX:+PrintGCApplicationStoppedTime
> -Dcom.sun.management.jmxremote.ssl=false
> -Dcom.sun.management.jmxremote.authenticate=false
> -Dcom.sun.management.jmxremote.port=18983
> -XX:OnOutOfMemoryError=/home/solr/solr-6.2.1/bin/oom_solr.sh
> -XX:NewSize=128m
> -XX:MaxNewSize=128m
> -XX:+UseG1GC
> -XX:+PerfDisableSharedMem
> -XX:+ParallelRefProcEnabled
> -XX:G1HeapRegionSize=8m
> -XX:MaxGCPauseMillis=250
> -XX:InitiatingHeapOccupancyPercent=75
> -XX:+UseLargePages
> -XX:+AggressiveOpts
> -Xmx32G
> -Xms32G
> -Xss256k
>
>
> Thanks & Regards
> Yasufumi.
>
> 2019年9月30日(月) 16:12 Deepak Goel <[hidden email]>:
>
> > Hello
> >
> > Can you please share the JVM heap settings in detail?
> >
> > Deepak
> >
> > On Mon, 30 Sep 2019, 11:15 Yasufumi Mizoguchi, <[hidden email]>
> > wrote:
> >
> > > Hi,
> > >
> > > I am trying some tests to confirm if single Solr instance can perform
> > over
> > > 1000 queries per second(!).
> > >
> > > But now, although CPU usage is 40% or so and iowait is almost 0%,
> > > throughput does not increase over 60 queries per second.
> > >
> > > I think there are some bottlenecks around Kernel, JVM, or Solr
> settings.
> > >
> > > The values we already checked and configured are followings.
> > >
> > > * Kernel:
> > > file descriptor
> > > net.ipv4.tcp_max_syn_backlog
> > > net.ipv4.tcp_syncookies
> > > net.core.somaxconn
> > > net.core.rmem_max
> > > net.core.wmem_max
> > > net.ipv4.tcp_rmem
> > > net.ipv4.tcp_wmem
> > >
> > > * JVM:
> > > Heap [ -> 32GB]
> > > G1GC settings
> > >
> > > * Solr:
> > > (Jetty) MaxThreads [ -> 20000]
> > >
> > >
> > > And the other info is as follows.
> > >
> > > CPU : 16 cores
> > > RAM : 128 GB
> > > Disk : SSD 500GB
> > > NIC : 10Gbps(maybe)
> > > OS : Ubuntu 14.04
> > > JVM : OpenJDK 1.8.0u191
> > > Solr : 6.2.1
> > > Index size : about 60GB
> > >
> > > Any insights will be appreciated.
> > >
> > > Thanks and regards,
> > > Yasufumi.
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Throughput does not increase in spite of low CPU usage

Jörn Franke
In reply to this post by Yasufumi Mizoguchi
Why do you need 1000 qps?


> Am 30.09.2019 um 07:45 schrieb Yasufumi Mizoguchi <[hidden email]>:
>
> Hi,
>
> I am trying some tests to confirm if single Solr instance can perform over
> 1000 queries per second(!).
>
> But now, although CPU usage is 40% or so and iowait is almost 0%,
> throughput does not increase over 60 queries per second.
>
> I think there are some bottlenecks around Kernel, JVM, or Solr settings.
>
> The values we already checked and configured are followings.
>
> * Kernel:
> file descriptor
> net.ipv4.tcp_max_syn_backlog
> net.ipv4.tcp_syncookies
> net.core.somaxconn
> net.core.rmem_max
> net.core.wmem_max
> net.ipv4.tcp_rmem
> net.ipv4.tcp_wmem
>
> * JVM:
> Heap [ -> 32GB]
> G1GC settings
>
> * Solr:
> (Jetty) MaxThreads [ -> 20000]
>
>
> And the other info is as follows.
>
> CPU : 16 cores
> RAM : 128 GB
> Disk : SSD 500GB
> NIC : 10Gbps(maybe)
> OS : Ubuntu 14.04
> JVM : OpenJDK 1.8.0u191
> Solr : 6.2.1
> Index size : about 60GB
>
> Any insights will be appreciated.
>
> Thanks and regards,
> Yasufumi.
Reply | Threaded
Open this post in threaded view
|

Re: Throughput does not increase in spite of low CPU usage

Paras Lehana
Besides what all have suggested, can you share your testing setup? Are you
using JMeter? I'm asking this to confirm that your setup is "actually
trying" to generate 1000 simultaneous threads and not bottlenecking the
process. I get 170 qps easily on a simpler server but I remember that I
could not go past 200 threads due to my testing PC limitations (I was
firing queries from my PC to the server).

On Tue, 1 Oct 2019 at 11:29, Jörn Franke <[hidden email]> wrote:

> Why do you need 1000 qps?
>
>
> > Am 30.09.2019 um 07:45 schrieb Yasufumi Mizoguchi <
> [hidden email]>:
> >
> > Hi,
> >
> > I am trying some tests to confirm if single Solr instance can perform
> over
> > 1000 queries per second(!).
> >
> > But now, although CPU usage is 40% or so and iowait is almost 0%,
> > throughput does not increase over 60 queries per second.
> >
> > I think there are some bottlenecks around Kernel, JVM, or Solr settings.
> >
> > The values we already checked and configured are followings.
> >
> > * Kernel:
> > file descriptor
> > net.ipv4.tcp_max_syn_backlog
> > net.ipv4.tcp_syncookies
> > net.core.somaxconn
> > net.core.rmem_max
> > net.core.wmem_max
> > net.ipv4.tcp_rmem
> > net.ipv4.tcp_wmem
> >
> > * JVM:
> > Heap [ -> 32GB]
> > G1GC settings
> >
> > * Solr:
> > (Jetty) MaxThreads [ -> 20000]
> >
> >
> > And the other info is as follows.
> >
> > CPU : 16 cores
> > RAM : 128 GB
> > Disk : SSD 500GB
> > NIC : 10Gbps(maybe)
> > OS : Ubuntu 14.04
> > JVM : OpenJDK 1.8.0u191
> > Solr : 6.2.1
> > Index size : about 60GB
> >
> > Any insights will be appreciated.
> >
> > Thanks and regards,
> > Yasufumi.
>


--
--
Regards,

*Paras Lehana* [65871]
Software Programmer, Auto-Suggest,
IndiaMART Intermesh Ltd.

8th Floor, Tower A, Advant-Navis Business Park, Sector 142,
Noida, UP, IN - 201303

Mob.: +91-9560911996
Work: 01203916600 | Extn:  *8173*

--
IMPORTANT: 
NEVER share your IndiaMART OTP/ Password with anyone.
Reply | Threaded
Open this post in threaded view
|

Re: Throughput does not increase in spite of low CPU usage

Yasufumi Mizoguchi
In reply to this post by Erick Erickson
Thank you for replying me.

Followings are current load test set up.

* Load test program : JUnit
* The number of Firing hosts : 6
* [Each host]ThreadGroup.num_threads : 200
* [Each host]ThreadGroup.ramp_time : 600
* [Each host]ThreadGroup.duration: 1800
* [Each host]ThreadGroup.delay: 0
* The number of sample queries: 20,000,000

And we confirmed that Jetty threads was increasing and reached the limit
(10000).
Therefore, we raised MaxThreads value.

I checked GC logs and found that it happened no major GC, and almost all
minor GC were finished by 200ms.

Cache hit rate is not checked now, but I think those are extremely low all
kinds of cache.
Because the number of sample query is big(20,000,000) compared to
queryResult and filter cache size(both 512) and there are few duplication
in fq and q parameter.
Those small cache size are intended, but I want to change those....

Thanks,
Yasufumi



2019年9月30日(月) 20:49 Erick Erickson <[hidden email]>:

> The most basic question is how you are load-testing it? Assuming you have
> some kind of client firing queries at Solr, keep adding threads so Solr is
> handling more and more queries in parallel. If you start to see the
> response time at the client get longer _and_ the  QTime in Solr’s response
> stays about the same, then the queries are queueing up and you need to see
> about increasing the Jetty threads handling queries.
>
> Second is whether you’re hitting GC pauses, look at the GC logs,
> especially for “stop the world” pauses. This is unlikely as you’re still
> getting 60 qps, but something to check.
>
> Setting your heap to 31G is good advice, but it won’t dramatically
> increase the throughput I’d guess.
>
> If your I/O isn’t very high, then your index is mostly memory-resident. A
> general bit of tuning advice is to _reduce_ the heap size, leaving OS
> memory for the index. See Uwe’s blog:
> https://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html.
> There’s a sweet spot between having too much heap and too little, and
> unfortunately you have to experiment to find out.
>
> But given the numbers you’re talking about, you won’t be getting 1,000 QPS
> on a single box and you’ll have to scale out with replicas to hit that
> number. Getting all the QPS you can out of the box is important, of course.
> Do be careful to use enough different queries that you don’t get them from
> the queryResultCache. I had one client who was thrilled they were getting
> 3ms response times…. by firing the same query over and over and hitting the
> queryResultCache 99.9999% of the time ;).
>
> Best,
> Erick
>
> > On Sep 30, 2019, at 4:28 AM, Yasufumi Mizoguchi <[hidden email]>
> wrote:
> >
> > Hi, Ere.
> >
> > Thank you for valuable feedback.
> > I will try Xmx31G and Xms31G instead of current ones.
> >
> > Thanks and Regards,
> > Yasufumi.
> >
> > 2019年9月30日(月) 17:19 Ere Maijala <[hidden email]>:
> >
> >> Just a side note: -Xmx32G is really bad for performance as it forces
> >> Java to use non-compressed pointers. You'll actually get better results
> >> with -Xmx31G. For more information, see e.g.
> >>
> >>
> https://blog.codecentric.de/en/2014/02/35gb-heap-less-32gb-java-jvm-memory-oddities/
> >>
> >> Regards,
> >> Ere
> >>
> >> Yasufumi Mizoguchi kirjoitti 30.9.2019 klo 11.05:
> >>> Hi, Deepak.
> >>> Thank you for replying me.
> >>>
> >>> JVM settings from solr.in.sh file are as follows. (Sorry, I could not
> >> show
> >>> all due to our policy)
> >>>
> >>> -verbose:gc
> >>> -XX:+PrintHeapAtGC
> >>> -XX:+PrintGCDetails
> >>> -XX:+PrintGCDateStamps
> >>> -XX:+PrintGCTimeStamps
> >>> -XX:+PrintTenuringDistribution
> >>> -XX:+PrintGCApplicationStoppedTime
> >>> -Dcom.sun.management.jmxremote.ssl=false
> >>> -Dcom.sun.management.jmxremote.authenticate=false
> >>> -Dcom.sun.management.jmxremote.port=18983
> >>> -XX:OnOutOfMemoryError=/home/solr/solr-6.2.1/bin/oom_solr.sh
> >>> -XX:NewSize=128m
> >>> -XX:MaxNewSize=128m
> >>> -XX:+UseG1GC
> >>> -XX:+PerfDisableSharedMem
> >>> -XX:+ParallelRefProcEnabled
> >>> -XX:G1HeapRegionSize=8m
> >>> -XX:MaxGCPauseMillis=250
> >>> -XX:InitiatingHeapOccupancyPercent=75
> >>> -XX:+UseLargePages
> >>> -XX:+AggressiveOpts
> >>> -Xmx32G
> >>> -Xms32G
> >>> -Xss256k
> >>>
> >>>
> >>> Thanks & Regards
> >>> Yasufumi.
> >>>
> >>> 2019年9月30日(月) 16:12 Deepak Goel <[hidden email]>:
> >>>
> >>>> Hello
> >>>>
> >>>> Can you please share the JVM heap settings in detail?
> >>>>
> >>>> Deepak
> >>>>
> >>>> On Mon, 30 Sep 2019, 11:15 Yasufumi Mizoguchi, <
> [hidden email]>
> >>>> wrote:
> >>>>
> >>>>> Hi,
> >>>>>
> >>>>> I am trying some tests to confirm if single Solr instance can perform
> >>>> over
> >>>>> 1000 queries per second(!).
> >>>>>
> >>>>> But now, although CPU usage is 40% or so and iowait is almost 0%,
> >>>>> throughput does not increase over 60 queries per second.
> >>>>>
> >>>>> I think there are some bottlenecks around Kernel, JVM, or Solr
> >> settings.
> >>>>>
> >>>>> The values we already checked and configured are followings.
> >>>>>
> >>>>> * Kernel:
> >>>>> file descriptor
> >>>>> net.ipv4.tcp_max_syn_backlog
> >>>>> net.ipv4.tcp_syncookies
> >>>>> net.core.somaxconn
> >>>>> net.core.rmem_max
> >>>>> net.core.wmem_max
> >>>>> net.ipv4.tcp_rmem
> >>>>> net.ipv4.tcp_wmem
> >>>>>
> >>>>> * JVM:
> >>>>> Heap [ -> 32GB]
> >>>>> G1GC settings
> >>>>>
> >>>>> * Solr:
> >>>>> (Jetty) MaxThreads [ -> 20000]
> >>>>>
> >>>>>
> >>>>> And the other info is as follows.
> >>>>>
> >>>>> CPU : 16 cores
> >>>>> RAM : 128 GB
> >>>>> Disk : SSD 500GB
> >>>>> NIC : 10Gbps(maybe)
> >>>>> OS : Ubuntu 14.04
> >>>>> JVM : OpenJDK 1.8.0u191
> >>>>> Solr : 6.2.1
> >>>>> Index size : about 60GB
> >>>>>
> >>>>> Any insights will be appreciated.
> >>>>>
> >>>>> Thanks and regards,
> >>>>> Yasufumi.
> >>>>>
> >>>>
> >>>
> >>
> >> --
> >> Ere Maijala
> >> Kansalliskirjasto / The National Library of Finland
> >>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Throughput does not increase in spite of low CPU usage

Yasufumi Mizoguchi
In reply to this post by Shawn Heisey-2
Thank you for your response.

Our index has about 25,000,000 docs. Is this quite difficult to achieve
1000 qps with single Solr instance even if all index files are cached on OS
page cache?

And as you pointed out, there are lots of minor GC during the test, I will
try to resize NewRatio.

Thanks,
Yasufumi.

2019年9月30日(月) 22:18 Shawn Heisey <[hidden email]>:

> On 9/29/2019 11:44 PM, Yasufumi Mizoguchi wrote:
> > I am trying some tests to confirm if single Solr instance can perform
> over
> > 1000 queries per second(!).
>
> In general, I would never expect a single instance to handle a large
> number of queries per second unless the index is REALLY small -- dozens
> or hundreds of very small documents.  A 60GB index definitely does not
> qualify.
>
> I don't think it will be possible to handle 1000 queries per second with
> a single server even on a really small index, but I've never actually
> tried.
>
> > But now, although CPU usage is 40% or so and iowait is almost 0%,
> > throughput does not increase over 60 queries per second.
>
> A query rate of 60 per second is pretty good with an index size of 60GB.
>   The low iowait would tend to confirm that the index is well cached by
> the OS.
>
> If you need to handle 1000 queries per second, you need more copies of
> your index on additional Solr servers, with something in the mix to
> perform load balancing.
>
> Some thoughts:
>
> With your -XX:MaxNewSize=128m setting, you are likely causing garbage
> collection to occur VERY frequently, which will slow things down.
> Solr's default GC settings include -XX:NewRatio=3 so that the new
> generation will be much larger than what you have set.  A program like
> Solr that allocates a lot of memory will need a fairly large new
> generation.
>
> I agree with the idea of setting the heap to 31GB.  Setting it to 31GB
> will actually leave more memory available to Solr than setting it to
> 32GB, because of the decreased pointer sizes.
>
> Definitely check what Erick mentioned.  If you're seeing what he
> described, adjusting how threads work might get you more throughput.
> But look into your new generation sizing first.
>
> Thanks,
> Shawn
>
Reply | Threaded
Open this post in threaded view
|

Re: Throughput does not increase in spite of low CPU usage

Yasufumi Mizoguchi
In reply to this post by Walter Underwood
Thank you for your response.

Now, we have no JVM monitoring.
But checking GC logs, I found no major GC during load test.
As you saying, heap size might be too large and I am planning to reduce
that.

Thanks,
Yasufumi

2019年9月30日(月) 23:19 Walter Underwood <[hidden email]>:

> 31G is still a very large heap. We use 8G for all of our different
> clusters.
>
> Do you have JVM monitoring? Look at the heap used after a major GC. Use
> that number, plus some extra, for the heap size.
>
> wunder
> Walter Underwood
> [hidden email]
> http://observer.wunderwood.org/  (my blog)
>
> > On Sep 30, 2019, at 1:28 AM, Yasufumi Mizoguchi <[hidden email]>
> wrote:
> >
> > Hi, Ere.
> >
> > Thank you for valuable feedback.
> > I will try Xmx31G and Xms31G instead of current ones.
> >
> > Thanks and Regards,
> > Yasufumi.
> >
> > 2019年9月30日(月) 17:19 Ere Maijala <[hidden email]>:
> >
> >> Just a side note: -Xmx32G is really bad for performance as it forces
> >> Java to use non-compressed pointers. You'll actually get better results
> >> with -Xmx31G. For more information, see e.g.
> >>
> >>
> https://blog.codecentric.de/en/2014/02/35gb-heap-less-32gb-java-jvm-memory-oddities/
> >>
> >> Regards,
> >> Ere
> >>
> >> Yasufumi Mizoguchi kirjoitti 30.9.2019 klo 11.05:
> >>> Hi, Deepak.
> >>> Thank you for replying me.
> >>>
> >>> JVM settings from solr.in.sh file are as follows. (Sorry, I could not
> >> show
> >>> all due to our policy)
> >>>
> >>> -verbose:gc
> >>> -XX:+PrintHeapAtGC
> >>> -XX:+PrintGCDetails
> >>> -XX:+PrintGCDateStamps
> >>> -XX:+PrintGCTimeStamps
> >>> -XX:+PrintTenuringDistribution
> >>> -XX:+PrintGCApplicationStoppedTime
> >>> -Dcom.sun.management.jmxremote.ssl=false
> >>> -Dcom.sun.management.jmxremote.authenticate=false
> >>> -Dcom.sun.management.jmxremote.port=18983
> >>> -XX:OnOutOfMemoryError=/home/solr/solr-6.2.1/bin/oom_solr.sh
> >>> -XX:NewSize=128m
> >>> -XX:MaxNewSize=128m
> >>> -XX:+UseG1GC
> >>> -XX:+PerfDisableSharedMem
> >>> -XX:+ParallelRefProcEnabled
> >>> -XX:G1HeapRegionSize=8m
> >>> -XX:MaxGCPauseMillis=250
> >>> -XX:InitiatingHeapOccupancyPercent=75
> >>> -XX:+UseLargePages
> >>> -XX:+AggressiveOpts
> >>> -Xmx32G
> >>> -Xms32G
> >>> -Xss256k
> >>>
> >>>
> >>> Thanks & Regards
> >>> Yasufumi.
> >>>
> >>> 2019年9月30日(月) 16:12 Deepak Goel <[hidden email]>:
> >>>
> >>>> Hello
> >>>>
> >>>> Can you please share the JVM heap settings in detail?
> >>>>
> >>>> Deepak
> >>>>
> >>>> On Mon, 30 Sep 2019, 11:15 Yasufumi Mizoguchi, <
> [hidden email]>
> >>>> wrote:
> >>>>
> >>>>> Hi,
> >>>>>
> >>>>> I am trying some tests to confirm if single Solr instance can perform
> >>>> over
> >>>>> 1000 queries per second(!).
> >>>>>
> >>>>> But now, although CPU usage is 40% or so and iowait is almost 0%,
> >>>>> throughput does not increase over 60 queries per second.
> >>>>>
> >>>>> I think there are some bottlenecks around Kernel, JVM, or Solr
> >> settings.
> >>>>>
> >>>>> The values we already checked and configured are followings.
> >>>>>
> >>>>> * Kernel:
> >>>>> file descriptor
> >>>>> net.ipv4.tcp_max_syn_backlog
> >>>>> net.ipv4.tcp_syncookies
> >>>>> net.core.somaxconn
> >>>>> net.core.rmem_max
> >>>>> net.core.wmem_max
> >>>>> net.ipv4.tcp_rmem
> >>>>> net.ipv4.tcp_wmem
> >>>>>
> >>>>> * JVM:
> >>>>> Heap [ -> 32GB]
> >>>>> G1GC settings
> >>>>>
> >>>>> * Solr:
> >>>>> (Jetty) MaxThreads [ -> 20000]
> >>>>>
> >>>>>
> >>>>> And the other info is as follows.
> >>>>>
> >>>>> CPU : 16 cores
> >>>>> RAM : 128 GB
> >>>>> Disk : SSD 500GB
> >>>>> NIC : 10Gbps(maybe)
> >>>>> OS : Ubuntu 14.04
> >>>>> JVM : OpenJDK 1.8.0u191
> >>>>> Solr : 6.2.1
> >>>>> Index size : about 60GB
> >>>>>
> >>>>> Any insights will be appreciated.
> >>>>>
> >>>>> Thanks and regards,
> >>>>> Yasufumi.
> >>>>>
> >>>>
> >>>
> >>
> >> --
> >> Ere Maijala
> >> Kansalliskirjasto / The National Library of Finland
> >>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Throughput does not increase in spite of low CPU usage

Yasufumi Mizoguchi
In reply to this post by Yasufumi Mizoguchi
Ah, sorry.

Not JUnit, we use JMeter.

Thanks,
Yasufumi

2019年10月1日(火) 19:08 Yasufumi Mizoguchi <[hidden email]>:

> Thank you for replying me.
>
> Followings are current load test set up.
>
> * Load test program : JUnit
> * The number of Firing hosts : 6
> * [Each host]ThreadGroup.num_threads : 200
> * [Each host]ThreadGroup.ramp_time : 600
> * [Each host]ThreadGroup.duration: 1800
> * [Each host]ThreadGroup.delay: 0
> * The number of sample queries: 20,000,000
>
> And we confirmed that Jetty threads was increasing and reached the limit
> (10000).
> Therefore, we raised MaxThreads value.
>
> I checked GC logs and found that it happened no major GC, and almost all
> minor GC were finished by 200ms.
>
> Cache hit rate is not checked now, but I think those are extremely low all
> kinds of cache.
> Because the number of sample query is big(20,000,000) compared to
> queryResult and filter cache size(both 512) and there are few duplication
> in fq and q parameter.
> Those small cache size are intended, but I want to change those....
>
> Thanks,
> Yasufumi
>
>
>
> 2019年9月30日(月) 20:49 Erick Erickson <[hidden email]>:
>
>> The most basic question is how you are load-testing it? Assuming you have
>> some kind of client firing queries at Solr, keep adding threads so Solr is
>> handling more and more queries in parallel. If you start to see the
>> response time at the client get longer _and_ the  QTime in Solr’s response
>> stays about the same, then the queries are queueing up and you need to see
>> about increasing the Jetty threads handling queries.
>>
>> Second is whether you’re hitting GC pauses, look at the GC logs,
>> especially for “stop the world” pauses. This is unlikely as you’re still
>> getting 60 qps, but something to check.
>>
>> Setting your heap to 31G is good advice, but it won’t dramatically
>> increase the throughput I’d guess.
>>
>> If your I/O isn’t very high, then your index is mostly memory-resident. A
>> general bit of tuning advice is to _reduce_ the heap size, leaving OS
>> memory for the index. See Uwe’s blog:
>> https://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html.
>> There’s a sweet spot between having too much heap and too little, and
>> unfortunately you have to experiment to find out.
>>
>> But given the numbers you’re talking about, you won’t be getting 1,000
>> QPS on a single box and you’ll have to scale out with replicas to hit that
>> number. Getting all the QPS you can out of the box is important, of course.
>> Do be careful to use enough different queries that you don’t get them from
>> the queryResultCache. I had one client who was thrilled they were getting
>> 3ms response times…. by firing the same query over and over and hitting the
>> queryResultCache 99.9999% of the time ;).
>>
>> Best,
>> Erick
>>
>> > On Sep 30, 2019, at 4:28 AM, Yasufumi Mizoguchi <[hidden email]>
>> wrote:
>> >
>> > Hi, Ere.
>> >
>> > Thank you for valuable feedback.
>> > I will try Xmx31G and Xms31G instead of current ones.
>> >
>> > Thanks and Regards,
>> > Yasufumi.
>> >
>> > 2019年9月30日(月) 17:19 Ere Maijala <[hidden email]>:
>> >
>> >> Just a side note: -Xmx32G is really bad for performance as it forces
>> >> Java to use non-compressed pointers. You'll actually get better results
>> >> with -Xmx31G. For more information, see e.g.
>> >>
>> >>
>> https://blog.codecentric.de/en/2014/02/35gb-heap-less-32gb-java-jvm-memory-oddities/
>> >>
>> >> Regards,
>> >> Ere
>> >>
>> >> Yasufumi Mizoguchi kirjoitti 30.9.2019 klo 11.05:
>> >>> Hi, Deepak.
>> >>> Thank you for replying me.
>> >>>
>> >>> JVM settings from solr.in.sh file are as follows. (Sorry, I could not
>> >> show
>> >>> all due to our policy)
>> >>>
>> >>> -verbose:gc
>> >>> -XX:+PrintHeapAtGC
>> >>> -XX:+PrintGCDetails
>> >>> -XX:+PrintGCDateStamps
>> >>> -XX:+PrintGCTimeStamps
>> >>> -XX:+PrintTenuringDistribution
>> >>> -XX:+PrintGCApplicationStoppedTime
>> >>> -Dcom.sun.management.jmxremote.ssl=false
>> >>> -Dcom.sun.management.jmxremote.authenticate=false
>> >>> -Dcom.sun.management.jmxremote.port=18983
>> >>> -XX:OnOutOfMemoryError=/home/solr/solr-6.2.1/bin/oom_solr.sh
>> >>> -XX:NewSize=128m
>> >>> -XX:MaxNewSize=128m
>> >>> -XX:+UseG1GC
>> >>> -XX:+PerfDisableSharedMem
>> >>> -XX:+ParallelRefProcEnabled
>> >>> -XX:G1HeapRegionSize=8m
>> >>> -XX:MaxGCPauseMillis=250
>> >>> -XX:InitiatingHeapOccupancyPercent=75
>> >>> -XX:+UseLargePages
>> >>> -XX:+AggressiveOpts
>> >>> -Xmx32G
>> >>> -Xms32G
>> >>> -Xss256k
>> >>>
>> >>>
>> >>> Thanks & Regards
>> >>> Yasufumi.
>> >>>
>> >>> 2019年9月30日(月) 16:12 Deepak Goel <[hidden email]>:
>> >>>
>> >>>> Hello
>> >>>>
>> >>>> Can you please share the JVM heap settings in detail?
>> >>>>
>> >>>> Deepak
>> >>>>
>> >>>> On Mon, 30 Sep 2019, 11:15 Yasufumi Mizoguchi, <
>> [hidden email]>
>> >>>> wrote:
>> >>>>
>> >>>>> Hi,
>> >>>>>
>> >>>>> I am trying some tests to confirm if single Solr instance can
>> perform
>> >>>> over
>> >>>>> 1000 queries per second(!).
>> >>>>>
>> >>>>> But now, although CPU usage is 40% or so and iowait is almost 0%,
>> >>>>> throughput does not increase over 60 queries per second.
>> >>>>>
>> >>>>> I think there are some bottlenecks around Kernel, JVM, or Solr
>> >> settings.
>> >>>>>
>> >>>>> The values we already checked and configured are followings.
>> >>>>>
>> >>>>> * Kernel:
>> >>>>> file descriptor
>> >>>>> net.ipv4.tcp_max_syn_backlog
>> >>>>> net.ipv4.tcp_syncookies
>> >>>>> net.core.somaxconn
>> >>>>> net.core.rmem_max
>> >>>>> net.core.wmem_max
>> >>>>> net.ipv4.tcp_rmem
>> >>>>> net.ipv4.tcp_wmem
>> >>>>>
>> >>>>> * JVM:
>> >>>>> Heap [ -> 32GB]
>> >>>>> G1GC settings
>> >>>>>
>> >>>>> * Solr:
>> >>>>> (Jetty) MaxThreads [ -> 20000]
>> >>>>>
>> >>>>>
>> >>>>> And the other info is as follows.
>> >>>>>
>> >>>>> CPU : 16 cores
>> >>>>> RAM : 128 GB
>> >>>>> Disk : SSD 500GB
>> >>>>> NIC : 10Gbps(maybe)
>> >>>>> OS : Ubuntu 14.04
>> >>>>> JVM : OpenJDK 1.8.0u191
>> >>>>> Solr : 6.2.1
>> >>>>> Index size : about 60GB
>> >>>>>
>> >>>>> Any insights will be appreciated.
>> >>>>>
>> >>>>> Thanks and regards,
>> >>>>> Yasufumi.
>> >>>>>
>> >>>>
>> >>>
>> >>
>> >> --
>> >> Ere Maijala
>> >> Kansalliskirjasto / The National Library of Finland
>> >>
>>
>>
Reply | Threaded
Open this post in threaded view
|

Re: Throughput does not increase in spite of low CPU usage

Yasufumi Mizoguchi
In reply to this post by Deepak Goel
Thank you for replying me.

I will try to resize NewRatio.

Thanks,
Yasufumi.

2019年10月1日(火) 11:19 Deepak Goel <[hidden email]>:

> Hello
>
> Can you please try increasing 'new size' and 'max new size' to 1GB+?
>
> Deepak
>
> On Mon, 30 Sep 2019, 13:35 Yasufumi Mizoguchi, <[hidden email]>
> wrote:
>
> > Hi, Deepak.
> > Thank you for replying me.
> >
> > JVM settings from solr.in.sh file are as follows. (Sorry, I could not
> show
> > all due to our policy)
> >
> > -verbose:gc
> > -XX:+PrintHeapAtGC
> > -XX:+PrintGCDetails
> > -XX:+PrintGCDateStamps
> > -XX:+PrintGCTimeStamps
> > -XX:+PrintTenuringDistribution
> > -XX:+PrintGCApplicationStoppedTime
> > -Dcom.sun.management.jmxremote.ssl=false
> > -Dcom.sun.management.jmxremote.authenticate=false
> > -Dcom.sun.management.jmxremote.port=18983
> > -XX:OnOutOfMemoryError=/home/solr/solr-6.2.1/bin/oom_solr.sh
> > -XX:NewSize=128m
> > -XX:MaxNewSize=128m
> > -XX:+UseG1GC
> > -XX:+PerfDisableSharedMem
> > -XX:+ParallelRefProcEnabled
> > -XX:G1HeapRegionSize=8m
> > -XX:MaxGCPauseMillis=250
> > -XX:InitiatingHeapOccupancyPercent=75
> > -XX:+UseLargePages
> > -XX:+AggressiveOpts
> > -Xmx32G
> > -Xms32G
> > -Xss256k
> >
> >
> > Thanks & Regards
> > Yasufumi.
> >
> > 2019年9月30日(月) 16:12 Deepak Goel <[hidden email]>:
> >
> > > Hello
> > >
> > > Can you please share the JVM heap settings in detail?
> > >
> > > Deepak
> > >
> > > On Mon, 30 Sep 2019, 11:15 Yasufumi Mizoguchi, <[hidden email]
> >
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > I am trying some tests to confirm if single Solr instance can perform
> > > over
> > > > 1000 queries per second(!).
> > > >
> > > > But now, although CPU usage is 40% or so and iowait is almost 0%,
> > > > throughput does not increase over 60 queries per second.
> > > >
> > > > I think there are some bottlenecks around Kernel, JVM, or Solr
> > settings.
> > > >
> > > > The values we already checked and configured are followings.
> > > >
> > > > * Kernel:
> > > > file descriptor
> > > > net.ipv4.tcp_max_syn_backlog
> > > > net.ipv4.tcp_syncookies
> > > > net.core.somaxconn
> > > > net.core.rmem_max
> > > > net.core.wmem_max
> > > > net.ipv4.tcp_rmem
> > > > net.ipv4.tcp_wmem
> > > >
> > > > * JVM:
> > > > Heap [ -> 32GB]
> > > > G1GC settings
> > > >
> > > > * Solr:
> > > > (Jetty) MaxThreads [ -> 20000]
> > > >
> > > >
> > > > And the other info is as follows.
> > > >
> > > > CPU : 16 cores
> > > > RAM : 128 GB
> > > > Disk : SSD 500GB
> > > > NIC : 10Gbps(maybe)
> > > > OS : Ubuntu 14.04
> > > > JVM : OpenJDK 1.8.0u191
> > > > Solr : 6.2.1
> > > > Index size : about 60GB
> > > >
> > > > Any insights will be appreciated.
> > > >
> > > > Thanks and regards,
> > > > Yasufumi.
> > > >
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Throughput does not increase in spite of low CPU usage

Paras Lehana
In reply to this post by Yasufumi Mizoguchi
Hi Yasufumi,

Followings are current load test set up.


Did you try decreasing ramp_time and increasing num_threads or Firing
hosts? Out of 1800 secs, you are giving 200 threads a maximum of 600
seconds to get ready. I don't increase with this value when I want to test
parallel requests. In your case, the answer could as simple as that the
testing threads itself are not capable of requesting more than 60 qps. Do
confirm this bottleneck by playing with JMeter values.

Cache hit rate is not checked now. Those small cache size are intended, but
> I want to change those.


I suggest you to check the cache stat in Solr Dashboard > Select Core >
Plugins/Stats > Cache. I'm assuming that all of your 20M queries are unique
though they could still use documentCache. Why not just comment out Cache
settings in solrconfig.xml?



On Tue, 1 Oct 2019 at 15:39, Yasufumi Mizoguchi <[hidden email]>
wrote:

> Thank you for replying me.
>
> Followings are current load test set up.
>
> * Load test program : JUnit
> * The number of Firing hosts : 6
> * [Each host]ThreadGroup.num_threads : 200
> * [Each host]ThreadGroup.ramp_time : 600
> * [Each host]ThreadGroup.duration: 1800
> * [Each host]ThreadGroup.delay: 0
> * The number of sample queries: 20,000,000
>
> And we confirmed that Jetty threads was increasing and reached the limit
> (10000).
> Therefore, we raised MaxThreads value.
>
> I checked GC logs and found that it happened no major GC, and almost all
> minor GC were finished by 200ms.
>
> Cache hit rate is not checked now, but I think those are extremely low all
> kinds of cache.
> Because the number of sample query is big(20,000,000) compared to
> queryResult and filter cache size(both 512) and there are few duplication
> in fq and q parameter.
> Those small cache size are intended, but I want to change those....
>
> Thanks,
> Yasufumi
>
>
>
> 2019年9月30日(月) 20:49 Erick Erickson <[hidden email]>:
>
> > The most basic question is how you are load-testing it? Assuming you have
> > some kind of client firing queries at Solr, keep adding threads so Solr
> is
> > handling more and more queries in parallel. If you start to see the
> > response time at the client get longer _and_ the  QTime in Solr’s
> response
> > stays about the same, then the queries are queueing up and you need to
> see
> > about increasing the Jetty threads handling queries.
> >
> > Second is whether you’re hitting GC pauses, look at the GC logs,
> > especially for “stop the world” pauses. This is unlikely as you’re still
> > getting 60 qps, but something to check.
> >
> > Setting your heap to 31G is good advice, but it won’t dramatically
> > increase the throughput I’d guess.
> >
> > If your I/O isn’t very high, then your index is mostly memory-resident. A
> > general bit of tuning advice is to _reduce_ the heap size, leaving OS
> > memory for the index. See Uwe’s blog:
> > https://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
> .
> > There’s a sweet spot between having too much heap and too little, and
> > unfortunately you have to experiment to find out.
> >
> > But given the numbers you’re talking about, you won’t be getting 1,000
> QPS
> > on a single box and you’ll have to scale out with replicas to hit that
> > number. Getting all the QPS you can out of the box is important, of
> course.
> > Do be careful to use enough different queries that you don’t get them
> from
> > the queryResultCache. I had one client who was thrilled they were getting
> > 3ms response times…. by firing the same query over and over and hitting
> the
> > queryResultCache 99.9999% of the time ;).
> >
> > Best,
> > Erick
> >
> > > On Sep 30, 2019, at 4:28 AM, Yasufumi Mizoguchi <
> [hidden email]>
> > wrote:
> > >
> > > Hi, Ere.
> > >
> > > Thank you for valuable feedback.
> > > I will try Xmx31G and Xms31G instead of current ones.
> > >
> > > Thanks and Regards,
> > > Yasufumi.
> > >
> > > 2019年9月30日(月) 17:19 Ere Maijala <[hidden email]>:
> > >
> > >> Just a side note: -Xmx32G is really bad for performance as it forces
> > >> Java to use non-compressed pointers. You'll actually get better
> results
> > >> with -Xmx31G. For more information, see e.g.
> > >>
> > >>
> >
> https://blog.codecentric.de/en/2014/02/35gb-heap-less-32gb-java-jvm-memory-oddities/
> > >>
> > >> Regards,
> > >> Ere
> > >>
> > >> Yasufumi Mizoguchi kirjoitti 30.9.2019 klo 11.05:
> > >>> Hi, Deepak.
> > >>> Thank you for replying me.
> > >>>
> > >>> JVM settings from solr.in.sh file are as follows. (Sorry, I could
> not
> > >> show
> > >>> all due to our policy)
> > >>>
> > >>> -verbose:gc
> > >>> -XX:+PrintHeapAtGC
> > >>> -XX:+PrintGCDetails
> > >>> -XX:+PrintGCDateStamps
> > >>> -XX:+PrintGCTimeStamps
> > >>> -XX:+PrintTenuringDistribution
> > >>> -XX:+PrintGCApplicationStoppedTime
> > >>> -Dcom.sun.management.jmxremote.ssl=false
> > >>> -Dcom.sun.management.jmxremote.authenticate=false
> > >>> -Dcom.sun.management.jmxremote.port=18983
> > >>> -XX:OnOutOfMemoryError=/home/solr/solr-6.2.1/bin/oom_solr.sh
> > >>> -XX:NewSize=128m
> > >>> -XX:MaxNewSize=128m
> > >>> -XX:+UseG1GC
> > >>> -XX:+PerfDisableSharedMem
> > >>> -XX:+ParallelRefProcEnabled
> > >>> -XX:G1HeapRegionSize=8m
> > >>> -XX:MaxGCPauseMillis=250
> > >>> -XX:InitiatingHeapOccupancyPercent=75
> > >>> -XX:+UseLargePages
> > >>> -XX:+AggressiveOpts
> > >>> -Xmx32G
> > >>> -Xms32G
> > >>> -Xss256k
> > >>>
> > >>>
> > >>> Thanks & Regards
> > >>> Yasufumi.
> > >>>
> > >>> 2019年9月30日(月) 16:12 Deepak Goel <[hidden email]>:
> > >>>
> > >>>> Hello
> > >>>>
> > >>>> Can you please share the JVM heap settings in detail?
> > >>>>
> > >>>> Deepak
> > >>>>
> > >>>> On Mon, 30 Sep 2019, 11:15 Yasufumi Mizoguchi, <
> > [hidden email]>
> > >>>> wrote:
> > >>>>
> > >>>>> Hi,
> > >>>>>
> > >>>>> I am trying some tests to confirm if single Solr instance can
> perform
> > >>>> over
> > >>>>> 1000 queries per second(!).
> > >>>>>
> > >>>>> But now, although CPU usage is 40% or so and iowait is almost 0%,
> > >>>>> throughput does not increase over 60 queries per second.
> > >>>>>
> > >>>>> I think there are some bottlenecks around Kernel, JVM, or Solr
> > >> settings.
> > >>>>>
> > >>>>> The values we already checked and configured are followings.
> > >>>>>
> > >>>>> * Kernel:
> > >>>>> file descriptor
> > >>>>> net.ipv4.tcp_max_syn_backlog
> > >>>>> net.ipv4.tcp_syncookies
> > >>>>> net.core.somaxconn
> > >>>>> net.core.rmem_max
> > >>>>> net.core.wmem_max
> > >>>>> net.ipv4.tcp_rmem
> > >>>>> net.ipv4.tcp_wmem
> > >>>>>
> > >>>>> * JVM:
> > >>>>> Heap [ -> 32GB]
> > >>>>> G1GC settings
> > >>>>>
> > >>>>> * Solr:
> > >>>>> (Jetty) MaxThreads [ -> 20000]
> > >>>>>
> > >>>>>
> > >>>>> And the other info is as follows.
> > >>>>>
> > >>>>> CPU : 16 cores
> > >>>>> RAM : 128 GB
> > >>>>> Disk : SSD 500GB
> > >>>>> NIC : 10Gbps(maybe)
> > >>>>> OS : Ubuntu 14.04
> > >>>>> JVM : OpenJDK 1.8.0u191
> > >>>>> Solr : 6.2.1
> > >>>>> Index size : about 60GB
> > >>>>>
> > >>>>> Any insights will be appreciated.
> > >>>>>
> > >>>>> Thanks and regards,
> > >>>>> Yasufumi.
> > >>>>>
> > >>>>
> > >>>
> > >>
> > >> --
> > >> Ere Maijala
> > >> Kansalliskirjasto / The National Library of Finland
> > >>
> >
> >
>


--
--
Regards,

*Paras Lehana* [65871]
Software Programmer, Auto-Suggest,
IndiaMART Intermesh Ltd.

8th Floor, Tower A, Advant-Navis Business Park, Sector 142,
Noida, UP, IN - 201303

Mob.: +91-9560911996
Work: 01203916600 | Extn:  *8173*

--
IMPORTANT: 
NEVER share your IndiaMART OTP/ Password with anyone.
Reply | Threaded
Open this post in threaded view
|

Re: Throughput does not increase in spite of low CPU usage

Yasufumi Mizoguchi
In reply to this post by Jörn Franke
It is difficult to answer that for me.

My customer requested me to achieve 1000 qps with single Solr.

Thanks,
Yasufumi.

2019年10月1日(火) 14:59 Jörn Franke <[hidden email]>:

> Why do you need 1000 qps?
>
>
> > Am 30.09.2019 um 07:45 schrieb Yasufumi Mizoguchi <
> [hidden email]>:
> >
> > Hi,
> >
> > I am trying some tests to confirm if single Solr instance can perform
> over
> > 1000 queries per second(!).
> >
> > But now, although CPU usage is 40% or so and iowait is almost 0%,
> > throughput does not increase over 60 queries per second.
> >
> > I think there are some bottlenecks around Kernel, JVM, or Solr settings.
> >
> > The values we already checked and configured are followings.
> >
> > * Kernel:
> > file descriptor
> > net.ipv4.tcp_max_syn_backlog
> > net.ipv4.tcp_syncookies
> > net.core.somaxconn
> > net.core.rmem_max
> > net.core.wmem_max
> > net.ipv4.tcp_rmem
> > net.ipv4.tcp_wmem
> >
> > * JVM:
> > Heap [ -> 32GB]
> > G1GC settings
> >
> > * Solr:
> > (Jetty) MaxThreads [ -> 20000]
> >
> >
> > And the other info is as follows.
> >
> > CPU : 16 cores
> > RAM : 128 GB
> > Disk : SSD 500GB
> > NIC : 10Gbps(maybe)
> > OS : Ubuntu 14.04
> > JVM : OpenJDK 1.8.0u191
> > Solr : 6.2.1
> > Index size : about 60GB
> >
> > Any insights will be appreciated.
> >
> > Thanks and regards,
> > Yasufumi.
>
Reply | Threaded
Open this post in threaded view
|

Re: Throughput does not increase in spite of low CPU usage

Yasufumi Mizoguchi
In reply to this post by Paras Lehana
Hi Paras,

Thank you for your advice.

I will confirm JMeter's settings in addition to JVM options.

Yes, we are using documentCache and *after* finishing load test, we will
comment that out.
(By customer requests, we cannot update cache settings till the test
ends...)

Thanks,
Yasufumi

2019年10月1日(火) 19:38 Paras Lehana <[hidden email]>:

> Hi Yasufumi,
>
> Followings are current load test set up.
>
>
> Did you try decreasing ramp_time and increasing num_threads or Firing
> hosts? Out of 1800 secs, you are giving 200 threads a maximum of 600
> seconds to get ready. I don't increase with this value when I want to test
> parallel requests. In your case, the answer could as simple as that the
> testing threads itself are not capable of requesting more than 60 qps. Do
> confirm this bottleneck by playing with JMeter values.
>
> Cache hit rate is not checked now. Those small cache size are intended, but
> > I want to change those.
>
>
> I suggest you to check the cache stat in Solr Dashboard > Select Core >
> Plugins/Stats > Cache. I'm assuming that all of your 20M queries are unique
> though they could still use documentCache. Why not just comment out Cache
> settings in solrconfig.xml?
>
>
>
> On Tue, 1 Oct 2019 at 15:39, Yasufumi Mizoguchi <[hidden email]>
> wrote:
>
> > Thank you for replying me.
> >
> > Followings are current load test set up.
> >
> > * Load test program : JUnit
> > * The number of Firing hosts : 6
> > * [Each host]ThreadGroup.num_threads : 200
> > * [Each host]ThreadGroup.ramp_time : 600
> > * [Each host]ThreadGroup.duration: 1800
> > * [Each host]ThreadGroup.delay: 0
> > * The number of sample queries: 20,000,000
> >
> > And we confirmed that Jetty threads was increasing and reached the limit
> > (10000).
> > Therefore, we raised MaxThreads value.
> >
> > I checked GC logs and found that it happened no major GC, and almost all
> > minor GC were finished by 200ms.
> >
> > Cache hit rate is not checked now, but I think those are extremely low
> all
> > kinds of cache.
> > Because the number of sample query is big(20,000,000) compared to
> > queryResult and filter cache size(both 512) and there are few duplication
> > in fq and q parameter.
> > Those small cache size are intended, but I want to change those....
> >
> > Thanks,
> > Yasufumi
> >
> >
> >
> > 2019年9月30日(月) 20:49 Erick Erickson <[hidden email]>:
> >
> > > The most basic question is how you are load-testing it? Assuming you
> have
> > > some kind of client firing queries at Solr, keep adding threads so Solr
> > is
> > > handling more and more queries in parallel. If you start to see the
> > > response time at the client get longer _and_ the  QTime in Solr’s
> > response
> > > stays about the same, then the queries are queueing up and you need to
> > see
> > > about increasing the Jetty threads handling queries.
> > >
> > > Second is whether you’re hitting GC pauses, look at the GC logs,
> > > especially for “stop the world” pauses. This is unlikely as you’re
> still
> > > getting 60 qps, but something to check.
> > >
> > > Setting your heap to 31G is good advice, but it won’t dramatically
> > > increase the throughput I’d guess.
> > >
> > > If your I/O isn’t very high, then your index is mostly
> memory-resident. A
> > > general bit of tuning advice is to _reduce_ the heap size, leaving OS
> > > memory for the index. See Uwe’s blog:
> > >
> https://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
> > .
> > > There’s a sweet spot between having too much heap and too little, and
> > > unfortunately you have to experiment to find out.
> > >
> > > But given the numbers you’re talking about, you won’t be getting 1,000
> > QPS
> > > on a single box and you’ll have to scale out with replicas to hit that
> > > number. Getting all the QPS you can out of the box is important, of
> > course.
> > > Do be careful to use enough different queries that you don’t get them
> > from
> > > the queryResultCache. I had one client who was thrilled they were
> getting
> > > 3ms response times…. by firing the same query over and over and hitting
> > the
> > > queryResultCache 99.9999% of the time ;).
> > >
> > > Best,
> > > Erick
> > >
> > > > On Sep 30, 2019, at 4:28 AM, Yasufumi Mizoguchi <
> > [hidden email]>
> > > wrote:
> > > >
> > > > Hi, Ere.
> > > >
> > > > Thank you for valuable feedback.
> > > > I will try Xmx31G and Xms31G instead of current ones.
> > > >
> > > > Thanks and Regards,
> > > > Yasufumi.
> > > >
> > > > 2019年9月30日(月) 17:19 Ere Maijala <[hidden email]>:
> > > >
> > > >> Just a side note: -Xmx32G is really bad for performance as it forces
> > > >> Java to use non-compressed pointers. You'll actually get better
> > results
> > > >> with -Xmx31G. For more information, see e.g.
> > > >>
> > > >>
> > >
> >
> https://blog.codecentric.de/en/2014/02/35gb-heap-less-32gb-java-jvm-memory-oddities/
> > > >>
> > > >> Regards,
> > > >> Ere
> > > >>
> > > >> Yasufumi Mizoguchi kirjoitti 30.9.2019 klo 11.05:
> > > >>> Hi, Deepak.
> > > >>> Thank you for replying me.
> > > >>>
> > > >>> JVM settings from solr.in.sh file are as follows. (Sorry, I could
> > not
> > > >> show
> > > >>> all due to our policy)
> > > >>>
> > > >>> -verbose:gc
> > > >>> -XX:+PrintHeapAtGC
> > > >>> -XX:+PrintGCDetails
> > > >>> -XX:+PrintGCDateStamps
> > > >>> -XX:+PrintGCTimeStamps
> > > >>> -XX:+PrintTenuringDistribution
> > > >>> -XX:+PrintGCApplicationStoppedTime
> > > >>> -Dcom.sun.management.jmxremote.ssl=false
> > > >>> -Dcom.sun.management.jmxremote.authenticate=false
> > > >>> -Dcom.sun.management.jmxremote.port=18983
> > > >>> -XX:OnOutOfMemoryError=/home/solr/solr-6.2.1/bin/oom_solr.sh
> > > >>> -XX:NewSize=128m
> > > >>> -XX:MaxNewSize=128m
> > > >>> -XX:+UseG1GC
> > > >>> -XX:+PerfDisableSharedMem
> > > >>> -XX:+ParallelRefProcEnabled
> > > >>> -XX:G1HeapRegionSize=8m
> > > >>> -XX:MaxGCPauseMillis=250
> > > >>> -XX:InitiatingHeapOccupancyPercent=75
> > > >>> -XX:+UseLargePages
> > > >>> -XX:+AggressiveOpts
> > > >>> -Xmx32G
> > > >>> -Xms32G
> > > >>> -Xss256k
> > > >>>
> > > >>>
> > > >>> Thanks & Regards
> > > >>> Yasufumi.
> > > >>>
> > > >>> 2019年9月30日(月) 16:12 Deepak Goel <[hidden email]>:
> > > >>>
> > > >>>> Hello
> > > >>>>
> > > >>>> Can you please share the JVM heap settings in detail?
> > > >>>>
> > > >>>> Deepak
> > > >>>>
> > > >>>> On Mon, 30 Sep 2019, 11:15 Yasufumi Mizoguchi, <
> > > [hidden email]>
> > > >>>> wrote:
> > > >>>>
> > > >>>>> Hi,
> > > >>>>>
> > > >>>>> I am trying some tests to confirm if single Solr instance can
> > perform
> > > >>>> over
> > > >>>>> 1000 queries per second(!).
> > > >>>>>
> > > >>>>> But now, although CPU usage is 40% or so and iowait is almost 0%,
> > > >>>>> throughput does not increase over 60 queries per second.
> > > >>>>>
> > > >>>>> I think there are some bottlenecks around Kernel, JVM, or Solr
> > > >> settings.
> > > >>>>>
> > > >>>>> The values we already checked and configured are followings.
> > > >>>>>
> > > >>>>> * Kernel:
> > > >>>>> file descriptor
> > > >>>>> net.ipv4.tcp_max_syn_backlog
> > > >>>>> net.ipv4.tcp_syncookies
> > > >>>>> net.core.somaxconn
> > > >>>>> net.core.rmem_max
> > > >>>>> net.core.wmem_max
> > > >>>>> net.ipv4.tcp_rmem
> > > >>>>> net.ipv4.tcp_wmem
> > > >>>>>
> > > >>>>> * JVM:
> > > >>>>> Heap [ -> 32GB]
> > > >>>>> G1GC settings
> > > >>>>>
> > > >>>>> * Solr:
> > > >>>>> (Jetty) MaxThreads [ -> 20000]
> > > >>>>>
> > > >>>>>
> > > >>>>> And the other info is as follows.
> > > >>>>>
> > > >>>>> CPU : 16 cores
> > > >>>>> RAM : 128 GB
> > > >>>>> Disk : SSD 500GB
> > > >>>>> NIC : 10Gbps(maybe)
> > > >>>>> OS : Ubuntu 14.04
> > > >>>>> JVM : OpenJDK 1.8.0u191
> > > >>>>> Solr : 6.2.1
> > > >>>>> Index size : about 60GB
> > > >>>>>
> > > >>>>> Any insights will be appreciated.
> > > >>>>>
> > > >>>>> Thanks and regards,
> > > >>>>> Yasufumi.
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>
> > > >> --
> > > >> Ere Maijala
> > > >> Kansalliskirjasto / The National Library of Finland
> > > >>
> > >
> > >
> >
>
>
> --
> --
> Regards,
>
> *Paras Lehana* [65871]
> Software Programmer, Auto-Suggest,
> IndiaMART Intermesh Ltd.
>
> 8th Floor, Tower A, Advant-Navis Business Park, Sector 142,
> Noida, UP, IN - 201303
>
> Mob.: +91-9560911996
> Work: 01203916600 | Extn:  *8173*
>
> --
> IMPORTANT:
> NEVER share your IndiaMART OTP/ Password with anyone.
>
Reply | Threaded
Open this post in threaded view
|

Re: Throughput does not increase in spite of low CPU usage

Toke Eskildsen-2
In reply to this post by Yasufumi Mizoguchi
On Tue, 2019-10-01 at 19:08 +0900, Yasufumi Mizoguchi wrote:

> * The number of Firing hosts : 6
> * [Each host]ThreadGroup.num_threads : 200
> * [Each host]ThreadGroup.ramp_time : 600
> * [Each host]ThreadGroup.duration: 1800
> * [Each host]ThreadGroup.delay: 0
> * The number of sample queries: 20,000,000
>
> And we confirmed that Jetty threads was increasing and reached the
> limit (10000).
> Therefore, we raised MaxThreads value.

You have a server with 16 CPU cores and you're running more than 10K
threads on it: That's a sure way to get lower throughput as the system
struggles to switch between the many threads. You would probably get
higher throughput by limiting the number of concurrent threads (for
once it seems that Erick and I are in a disagreement).

It's easy to try: Just set num_threads for each ThreadGroup to 20 and
see if your QPS rises above the 60 you have now.

Still, there's something off: 6*200 threads = 1200. That should not
blow Jetty thread allocation beyond 10K. Maybe your index is sharded?
that would explain extra thread allocation. If so, how many shards do
you have and have you tries running with a single shard? Single shard
indexes maximizes throughput at the possible cost of latency, so that
seems fitting for your requirements.

- Toke Eskildsen, Royal Danish Library


12