upgrading from solr4 to solr8 searches taking 4 to 10 times as long to return

classic Classic list List threaded Threaded
26 messages Options
12
Reply | Threaded
Open this post in threaded view
|

upgrading from solr4 to solr8 searches taking 4 to 10 times as long to return

Russell Bahr
Hi,
I am trying to replace our solr4 cluster with a solr 8.1.1 cluster and am
running into a problem where searches are taking way to long to respond.
The clusters are set up with the same number of servers, same number of
shards, and same number of replicas.  They are indexing the same documents,
but are returning results the results sometimes as much as 10 times faster
on the solr4 cluster than they are on the solr6 cluster.

for example,

solr4 cluster
30 machines
1 collection
6 shards
replication factor 5
sorl version 4.10.4
debian 9.9
8 vCPU
16 GB memory
approximately 18 million documents
config
*:* query across 10 times returning
[13234, 18714, 13384, 12966, 12192, 18420, 16592, 15691, 13373, 12458]
vs

solr8 cluster
30 machines
1 collection
6 shards
replication factor 5
sorl version 8.1.1
debian 9.9
8 vCPU
16 GB memory
approximately 18 million documents
*:* query across 10 times returning
[93359, 94263, 86949, 90747, 91171, 91588, 87921, 88632, 88035, 89137]

please let me know what else I can include that may help identify where the
slowness is occuring.

Thank you,

*Manzama*a MODERN GOVERNANCE company

Russell Bahr
Lead Infrastructure Engineer

USA & CAN Office: +1 (541) 306 3271
USA & CAN Support: +1 (541) 706 9393
UK Office & Support: +44 (0)203 282 1633
AUS Office & Support: +61 (0) 2 8417 2339

543 NW York Drive, Suite 100, Bend, OR 97703

LinkedIn <http://www.linkedin.com/company/manzama> | Twitter
<https://twitter.com/ManzamaInc> | Facebook
<http://www.facebook.com/manzamainc> | YouTube
<https://www.youtube.com/channel/UCBo3QoqewyNoo7HiT_BFuRw>
Reply | Threaded
Open this post in threaded view
|

Re: upgrading from solr4 to solr8 searches taking 4 to 10 times as long to return

Toke Eskildsen-2
Russell Bahr <[hidden email]> wrote:
> approximately 18 million documents
> *:* query across 10 times returning
> [13234, 18714, 13384, 12966, 12192, 18420, 16592, 15691, 13373, 12458]
>vs
> [93359, 94263, 86949, 90747, 91171, 91588, 87921, 88632, 88035, 89137]

Even the 12-18 seconds for Solr 4 is a long time, so I'm guessing that your queries are complex, you are asking for large result sets, heavy aggregations or something like that. Can you provide the full list of parameters in your request?

Related, 30 machines to handle 18 million documents sounds like a lot. Are you serving very large documents? How big is your index in bytes?

- Toke Eskildsen
Reply | Threaded
Open this post in threaded view
|

Re: upgrading from solr4 to solr8 searches taking 4 to 10 times as long to return

Russell Bahr
Hi Toke,

Yes, some of our queries are quite complex due to a lot of very specific
positive as well as negative boosts, however, the query that I ran as the
base test after we found our queries were taking so long is just "
http://solr.obscured.com:8990/solr/content/select?q=*%3A*&wt=json&indent=true
"

Our index sizes run between 12k to 36k per server



*Manzama*a MODERN GOVERNANCE company

Russell Bahr
Lead Infrastructure Engineer

USA & CAN Office: +1 (541) 306 3271
USA & CAN Support: +1 (541) 706 9393
UK Office & Support: +44 (0)203 282 1633
AUS Office & Support: +61 (0) 2 8417 2339

543 NW York Drive, Suite 100, Bend, OR 97703

LinkedIn <http://www.linkedin.com/company/manzama> | Twitter
<https://twitter.com/ManzamaInc> | Facebook
<http://www.facebook.com/manzamainc> | YouTube
<https://www.youtube.com/channel/UCBo3QoqewyNoo7HiT_BFuRw>


On Tue, Sep 3, 2019 at 10:54 AM Toke Eskildsen <[hidden email]> wrote:

> Russell Bahr <[hidden email]> wrote:
> > approximately 18 million documents
> > *:* query across 10 times returning
> > [13234, 18714, 13384, 12966, 12192, 18420, 16592, 15691, 13373, 12458]
> >vs
> > [93359, 94263, 86949, 90747, 91171, 91588, 87921, 88632, 88035, 89137]
>
> Even the 12-18 seconds for Solr 4 is a long time, so I'm guessing that
> your queries are complex, you are asking for large result sets, heavy
> aggregations or something like that. Can you provide the full list of
> parameters in your request?
>
> Related, 30 machines to handle 18 million documents sounds like a lot. Are
> you serving very large documents? How big is your index in bytes?
>
> - Toke Eskildsen
>
Reply | Threaded
Open this post in threaded view
|

Re: upgrading from solr4 to solr8 searches taking 4 to 10 times as long to return

Russell Bahr
Hi Toke,

Also, if it helps, the content on each server is between around 6.2Gb and
7.8Gb.

Thanks,

Russ

*Manzama*a MODERN GOVERNANCE company

Russell Bahr
Lead Infrastructure Engineer

USA & CAN Office: +1 (541) 306 3271
USA & CAN Support: +1 (541) 706 9393
UK Office & Support: +44 (0)203 282 1633
AUS Office & Support: +61 (0) 2 8417 2339

543 NW York Drive, Suite 100, Bend, OR 97703

LinkedIn <http://www.linkedin.com/company/manzama> | Twitter
<https://twitter.com/ManzamaInc> | Facebook
<http://www.facebook.com/manzamainc> | YouTube
<https://www.youtube.com/channel/UCBo3QoqewyNoo7HiT_BFuRw>


On Tue, Sep 3, 2019 at 12:22 PM Russell Bahr <[hidden email]> wrote:

> Hi Toke,
>
> Yes, some of our queries are quite complex due to a lot of very specific
> positive as well as negative boosts, however, the query that I ran as the
> base test after we found our queries were taking so long is just "
> http://solr.obscured.com:8990/solr/content/select?q=*%3A*&wt=json&indent=true
> "
>
> Our index sizes run between 12k to 36k per server
>
>
>
> *Manzama*a MODERN GOVERNANCE company
>
> Russell Bahr
> Lead Infrastructure Engineer
>
> USA & CAN Office: +1 (541) 306 3271
> USA & CAN Support: +1 (541) 706 9393
> UK Office & Support: +44 (0)203 282 1633
> AUS Office & Support: +61 (0) 2 8417 2339
>
> 543 NW York Drive, Suite 100, Bend, OR 97703
>
> LinkedIn <http://www.linkedin.com/company/manzama> | Twitter
> <https://twitter.com/ManzamaInc> | Facebook
> <http://www.facebook.com/manzamainc> | YouTube
> <https://www.youtube.com/channel/UCBo3QoqewyNoo7HiT_BFuRw>
>
>
> On Tue, Sep 3, 2019 at 10:54 AM Toke Eskildsen <[hidden email]> wrote:
>
>> Russell Bahr <[hidden email]> wrote:
>> > approximately 18 million documents
>> > *:* query across 10 times returning
>> > [13234, 18714, 13384, 12966, 12192, 18420, 16592, 15691, 13373, 12458]
>> >vs
>> > [93359, 94263, 86949, 90747, 91171, 91588, 87921, 88632, 88035, 89137]
>>
>> Even the 12-18 seconds for Solr 4 is a long time, so I'm guessing that
>> your queries are complex, you are asking for large result sets, heavy
>> aggregations or something like that. Can you provide the full list of
>> parameters in your request?
>>
>> Related, 30 machines to handle 18 million documents sounds like a lot.
>> Are you serving very large documents? How big is your index in bytes?
>>
>> - Toke Eskildsen
>>
>
Reply | Threaded
Open this post in threaded view
|

Re: upgrading from solr4 to solr8 searches taking 4 to 10 times as long to return

Shawn Heisey-2
In reply to this post by Russell Bahr
On 9/3/2019 1:22 PM, Russell Bahr wrote:
> Yes, some of our queries are quite complex due to a lot of very specific
> positive as well as negative boosts, however, the query that I ran as the
> base test after we found our queries were taking so long is just "
> http://solr.obscured.com:8990/solr/content/select?q=*%3A*&wt=json&indent=true
> "

If you've got an all documents query (the special *:* syntax) taking
many seconds to run, that is usually an indication of a serious
performance problem.  In most situations, I would expect the millisecond
response time to be only two or three digits, especially with your
relatively low document count.

The response times are worse with the new version, but I would call the
response times on the old version very slow as well.

The screenshot described here is someplace we can start looking for
performance issues:

https://cwiki.apache.org/confluence/display/solr/SolrPerformanceProblems#SolrPerformanceProblems-ProcesslistingonPOSIXoperatingsystems

Thanks,
Shawn
Reply | Threaded
Open this post in threaded view
|

Re: upgrading from solr4 to solr8 searches taking 4 to 10 times as long to return

Russell Bahr
Hi Shawn,
Here is a screenshot of one of the master nodes
solr4


solr8

Manzama
a MODERN GOVERNANCE company

Russell Bahr
Lead Infrastructure Engineer

USA & CAN Office: +1 (541) 306 3271 
USA & CAN Support: +1 (541) 706 9393 
UK Office & Support: +44 (0)203 282 1633
AUS Office & Support: +61 (0) 2 8417 2339

543 NW York Drive, Suite 100, Bend, OR 97703

LinkedIn | Twitter | Facebook | YouTube



On Tue, Sep 3, 2019 at 2:19 PM Shawn Heisey <[hidden email]> wrote:
On 9/3/2019 1:22 PM, Russell Bahr wrote:
> Yes, some of our queries are quite complex due to a lot of very specific
> positive as well as negative boosts, however, the query that I ran as the
> base test after we found our queries were taking so long is just "
> http://solr.obscured.com:8990/solr/content/select?q=*%3A*&wt=json&indent=true
> "

If you've got an all documents query (the special *:* syntax) taking
many seconds to run, that is usually an indication of a serious
performance problem.  In most situations, I would expect the millisecond
response time to be only two or three digits, especially with your
relatively low document count.

The response times are worse with the new version, but I would call the
response times on the old version very slow as well.

The screenshot described here is someplace we can start looking for
performance issues:

https://cwiki.apache.org/confluence/display/solr/SolrPerformanceProblems#SolrPerformanceProblems-ProcesslistingonPOSIXoperatingsystems

Thanks,
Shawn
Reply | Threaded
Open this post in threaded view
|

Re: upgrading from solr4 to solr8 searches taking 4 to 10 times as long to return

Shawn Heisey-2
On 9/3/2019 4:46 PM, Russell Bahr wrote:
> Hi Shawn,
> Here is a screenshot of one of the master nodes
> solr4
> Screen Shot 2019-09-03 at 3.37.08 PM.png
>
> solr8
> Screen Shot 2019-09-03 at 3.45.46 PM.png

Email attachments do not make it to the list.  I cannot see those
pictures.  You will need to use a file sharing website.  Dropbox is a
good choice.

https://www.dropbox.com/s/gx8gzmt3zcumf66/rbahr-list-attach.png?dl=0

Thanks,
Shawn
Reply | Threaded
Open this post in threaded view
|

Re: upgrading from solr4 to solr8 searches taking 4 to 10 times as long to return

Toke Eskildsen-2
In reply to this post by Russell Bahr
On Tue, 2019-09-03 at 12:35 -0700, Russell Bahr wrote:
> Also, if it helps, the content on each server is between around 6.2Gb
> and 7.8Gb.

We're still missing something here. The trivial query

http://solr.obscured.com:8990/solr/content/select?q=*%3A*&wt=json&indent=true
on such a modest index size & document count should not take seconds to
complete. So we must widen our questioning:

* What is your Xmx for the Solrs?
(If you use most of the physical memory for Java, there might be too
little left for OS-caching)

* Do you have a large number of stored or docValued fields?
(Solr 8 returns docValued fields per default while Solr 4 does not)

* How large is a response? One simple way to check is
curl '
http://solr.obscured.com:8990/solr/content/select?q=*%3A*&wt=json&indent=true' 
| wc -c
(Solr 4 and 8 compress stored content differently and a very large
result set requires more CPU power to uncompress in Solr 8 (but less
IO))

* Do you have any response related defaults in your solrconfig.xml,
such as faceting or grouping?
(You might be doing heavy aggregation even if you don't explicitly ask
for it)


- Toke Eskildsen, Royal Danish Library


Reply | Threaded
Open this post in threaded view
|

Re: upgrading from solr4 to solr8 searches taking 4 to 10 times as long to return

Russell Bahr
In reply to this post by Shawn Heisey-2
Hi Shawn,

Thank you for the feedback and advise.  I have loaded the 2 screenshots up
to drop box.  Here is the link.

https://www.dropbox.com/s/c5b41a61za0ojw7/solr4_Screen%20Shot%202019-09-03%20at%203.37.08%20PM.png?dl=0

Thank you,

*Manzama*a MODERN GOVERNANCE company

Russell Bahr
Lead Infrastructure Engineer

USA & CAN Office: +1 (541) 306 3271
USA & CAN Support: +1 (541) 706 9393
UK Office & Support: +44 (0)203 282 1633
AUS Office & Support: +61 (0) 2 8417 2339

543 NW York Drive, Suite 100, Bend, OR 97703

LinkedIn <http://www.linkedin.com/company/manzama> | Twitter
<https://twitter.com/ManzamaInc> | Facebook
<http://www.facebook.com/manzamainc> | YouTube
<https://www.youtube.com/channel/UCBo3QoqewyNoo7HiT_BFuRw>


On Tue, Sep 3, 2019 at 6:07 PM Shawn Heisey <[hidden email]> wrote:

> On 9/3/2019 4:46 PM, Russell Bahr wrote:
> > Hi Shawn,
> > Here is a screenshot of one of the master nodes
> > solr4
> > Screen Shot 2019-09-03 at 3.37.08 PM.png
> >
> > solr8
> > Screen Shot 2019-09-03 at 3.45.46 PM.png
>
> Email attachments do not make it to the list.  I cannot see those
> pictures.  You will need to use a file sharing website.  Dropbox is a
> good choice.
>
> https://www.dropbox.com/s/gx8gzmt3zcumf66/rbahr-list-attach.png?dl=0
>
> Thanks,
> Shawn
>
Reply | Threaded
Open this post in threaded view
|

Re: upgrading from solr4 to solr8 searches taking 4 to 10 times as long to return

Russell Bahr
In reply to this post by Toke Eskildsen-2
Hi Toke,
Please see below.  We reindexed the solr8 cluster to make sure it was up to
date with content.

* What is your Xmx for the Solrs? -
solr8

SOLR_JAVA_MEM="-Xms11235m -Xmx11235m" - 70% of OS Memory


solr4

export CATALINA_OPTS="-Xms11235m -Xmx11235m" - - 70% of OS Memory


(If you use most of the physical memory for Java, there might be too
little left for OS-caching)

* Do you have a large number of stored or docValued fields?
solr8
not sure if this would be considered a large amount or not.  Posted schema
in dropbox
https://www.dropbox.com/sh/hslknixd3azj7mi/AABnCXex_HInCvRz3kuKLwNna?dl=0
solr4
posted schema in dropbox
https://www.dropbox.com/sh/hslknixd3azj7mi/AABnCXex_HInCvRz3kuKLwNna?dl=0

(Solr 8 returns docValued fields per default while Solr 4 does not)

* How large is a response? One simple way to check is
curl '
http://solr.obscured.com:8990/solr/content/select?q=*%3A*&wt=json&indent=true
'
| wc -c

solr8

  % Total    % Received % Xferd  Average Speed   Time    Time     Time
Current

                                 Dload  Upload   Total   Spent    Left
Speed

100 18830  100 18830    0     0    245      0  0:01:16  0:01:16 --:--:--
5822

   18830

solr4

  % Total    % Received % Xferd  Average Speed   Time    Time     Time
Current

                                 Dload  Upload   Total   Spent    Left
Speed

100 13149    0 13149    0     0    807      0 --:--:--  0:00:16 --:--:--
3273

   13149



(Solr 4 and 8 compress stored content differently and a very large
result set requires more CPU power to uncompress in Solr 8 (but less
IO))

* Do you have any response related defaults in your solrconfig.xml,
such as faceting or grouping?

solr8
posted config
https://www.dropbox.com/sh/hslknixd3azj7mi/AABnCXex_HInCvRz3kuKLwNna?dl=0

solr4
posted config
https://www.dropbox.com/sh/hslknixd3azj7mi/AABnCXex_HInCvRz3kuKLwNna?dl=0

(You might be doing heavy aggregation even if you don't explicitly ask
for it)


*Manzama*a MODERN GOVERNANCE company

Russell Bahr
Lead Infrastructure Engineer

USA & CAN Office: +1 (541) 306 3271
USA & CAN Support: +1 (541) 706 9393
UK Office & Support: +44 (0)203 282 1633
AUS Office & Support: +61 (0) 2 8417 2339

543 NW York Drive, Suite 100, Bend, OR 97703

LinkedIn <http://www.linkedin.com/company/manzama> | Twitter
<https://twitter.com/ManzamaInc> | Facebook
<http://www.facebook.com/manzamainc> | YouTube
<https://www.youtube.com/channel/UCBo3QoqewyNoo7HiT_BFuRw>


On Wed, Sep 4, 2019 at 1:43 AM Toke Eskildsen <[hidden email]> wrote:

> On Tue, 2019-09-03 at 12:35 -0700, Russell Bahr wrote:
> > Also, if it helps, the content on each server is between around 6.2Gb
> > and 7.8Gb.
>
> We're still missing something here. The trivial query
>
>
> http://solr.obscured.com:8990/solr/content/select?q=*%3A*&wt=json&indent=true
> on such a modest index size & document count should not take seconds to
> complete. So we must widen our questioning:
>
> * What is your Xmx for the Solrs?
> (If you use most of the physical memory for Java, there might be too
> little left for OS-caching)
>
> * Do you have a large number of stored or docValued fields?
> (Solr 8 returns docValued fields per default while Solr 4 does not)
>
> * How large is a response? One simple way to check is
> curl '
>
> http://solr.obscured.com:8990/solr/content/select?q=*%3A*&wt=json&indent=true'
>
> | wc -c
> (Solr 4 and 8 compress stored content differently and a very large
> result set requires more CPU power to uncompress in Solr 8 (but less
> IO))
>
> * Do you have any response related defaults in your solrconfig.xml,
> such as faceting or grouping?
> (You might be doing heavy aggregation even if you don't explicitly ask
> for it)
>
>
> - Toke Eskildsen, Royal Danish Library
>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: upgrading from solr4 to solr8 searches taking 4 to 10 times as long to return

Russell Bahr
Hi Toke and Shawn,
Any thoughts on what I sent?
Thanks in advance,
Russ

*Manzama*a MODERN GOVERNANCE company

Russell Bahr
Lead Infrastructure Engineer

USA & CAN Office: +1 (541) 306 3271
USA & CAN Support: +1 (541) 706 9393
UK Office & Support: +44 (0)203 282 1633
AUS Office & Support: +61 (0) 2 8417 2339

543 NW York Drive, Suite 100, Bend, OR 97703

LinkedIn <http://www.linkedin.com/company/manzama> | Twitter
<https://twitter.com/ManzamaInc> | Facebook
<http://www.facebook.com/manzamainc> | YouTube
<https://www.youtube.com/channel/UCBo3QoqewyNoo7HiT_BFuRw>


On Wed, Sep 4, 2019 at 2:55 PM Russell Bahr <[hidden email]> wrote:

> Hi Toke,
> Please see below.  We reindexed the solr8 cluster to make sure it was up
> to date with content.
>
> * What is your Xmx for the Solrs? -
> solr8
>
> SOLR_JAVA_MEM="-Xms11235m -Xmx11235m" - 70% of OS Memory
>
>
> solr4
>
> export CATALINA_OPTS="-Xms11235m -Xmx11235m" - - 70% of OS Memory
>
>
> (If you use most of the physical memory for Java, there might be too
> little left for OS-caching)
>
> * Do you have a large number of stored or docValued fields?
> solr8
> not sure if this would be considered a large amount or not.  Posted schema
> in dropbox
> https://www.dropbox.com/sh/hslknixd3azj7mi/AABnCXex_HInCvRz3kuKLwNna?dl=0
> solr4
> posted schema in dropbox
> https://www.dropbox.com/sh/hslknixd3azj7mi/AABnCXex_HInCvRz3kuKLwNna?dl=0
>
> (Solr 8 returns docValued fields per default while Solr 4 does not)
>
> * How large is a response? One simple way to check is
> curl '
>
> http://solr.obscured.com:8990/solr/content/select?q=*%3A*&wt=json&indent=true
> '
> | wc -c
>
> solr8
>
>   % Total    % Received % Xferd  Average Speed   Time    Time     Time
> Current
>
>                                  Dload  Upload   Total   Spent    Left
> Speed
>
> 100 18830  100 18830    0     0    245      0  0:01:16  0:01:16 --:--:--
> 5822
>
>    18830
>
> solr4
>
>   % Total    % Received % Xferd  Average Speed   Time    Time     Time
> Current
>
>                                  Dload  Upload   Total   Spent    Left
> Speed
>
> 100 13149    0 13149    0     0    807      0 --:--:--  0:00:16 --:--:--
> 3273
>
>    13149
>
>
>
> (Solr 4 and 8 compress stored content differently and a very large
> result set requires more CPU power to uncompress in Solr 8 (but less
> IO))
>
> * Do you have any response related defaults in your solrconfig.xml,
> such as faceting or grouping?
>
> solr8
> posted config
> https://www.dropbox.com/sh/hslknixd3azj7mi/AABnCXex_HInCvRz3kuKLwNna?dl=0
>
> solr4
> posted config
> https://www.dropbox.com/sh/hslknixd3azj7mi/AABnCXex_HInCvRz3kuKLwNna?dl=0
>
> (You might be doing heavy aggregation even if you don't explicitly ask
> for it)
>
>
> *Manzama*a MODERN GOVERNANCE company
>
> Russell Bahr
> Lead Infrastructure Engineer
>
> USA & CAN Office: +1 (541) 306 3271
> USA & CAN Support: +1 (541) 706 9393
> UK Office & Support: +44 (0)203 282 1633
> AUS Office & Support: +61 (0) 2 8417 2339
>
> 543 NW York Drive, Suite 100, Bend, OR 97703
>
> LinkedIn <http://www.linkedin.com/company/manzama> | Twitter
> <https://twitter.com/ManzamaInc> | Facebook
> <http://www.facebook.com/manzamainc> | YouTube
> <https://www.youtube.com/channel/UCBo3QoqewyNoo7HiT_BFuRw>
>
>
> On Wed, Sep 4, 2019 at 1:43 AM Toke Eskildsen <[hidden email]> wrote:
>
>> On Tue, 2019-09-03 at 12:35 -0700, Russell Bahr wrote:
>> > Also, if it helps, the content on each server is between around 6.2Gb
>> > and 7.8Gb.
>>
>> We're still missing something here. The trivial query
>>
>>
>> http://solr.obscured.com:8990/solr/content/select?q=*%3A*&wt=json&indent=true
>> on such a modest index size & document count should not take seconds to
>> complete. So we must widen our questioning:
>>
>> * What is your Xmx for the Solrs?
>> (If you use most of the physical memory for Java, there might be too
>> little left for OS-caching)
>>
>> * Do you have a large number of stored or docValued fields?
>> (Solr 8 returns docValued fields per default while Solr 4 does not)
>>
>> * How large is a response? One simple way to check is
>> curl '
>>
>> http://solr.obscured.com:8990/solr/content/select?q=*%3A*&wt=json&indent=true'
>>
>> | wc -c
>> (Solr 4 and 8 compress stored content differently and a very large
>> result set requires more CPU power to uncompress in Solr 8 (but less
>> IO))
>>
>> * Do you have any response related defaults in your solrconfig.xml,
>> such as faceting or grouping?
>> (You might be doing heavy aggregation even if you don't explicitly ask
>> for it)
>>
>>
>> - Toke Eskildsen, Royal Danish Library
>>
>>
>>
Reply | Threaded
Open this post in threaded view
|

Re: upgrading from solr4 to solr8 searches taking 4 to 10 times as long to return

Shawn Heisey-2
In reply to this post by Russell Bahr
On 9/4/2019 12:48 PM, Russell Bahr wrote:
> Thank you for the feedback and advise.  I have loaded the 2 screenshots up
> to drop box.  Here is the link.
>
> https://www.dropbox.com/s/c5b41a61za0ojw7/solr4_Screen%20Shot%202019-09-03%20at%203.37.08%20PM.png?dl=0

Just one screenshot there.

Looking at that, I'm guessing that you have a heap size between 11 and
12 GB, and about 7GB of index data.  A little more than 3GB of your
system memory is being used for disk caching.  So you can fit about half
your index into the cache.

I would not ordinarily expect a situation like this to have such bad
performance, so there may be something going on that isn't
straightforward to see here.  One possible problem is that your install
actually needs a bigger heap than you have, so it's spending a lot of
time doing garbage collection.  If this is what's happening, you
probably are going to need more than 16GB total memory.

What is the total document count for that 7GB of index data?

Solr 4 likely isn't writing a GC log, but Solr 8 probably is.  Can you
share those logs?

Thanks,
Shawn
Reply | Threaded
Open this post in threaded view
|

Re: upgrading from solr4 to solr8 searches taking 4 to 10 times as long to return

Russell Bahr
Hi Shawn,
Sorry for the other link.  I figured out after I sent the first one how to
share the entire folder.  Please try this link and let me know if that
works.

https://www.dropbox.com/sh/hslknixd3azj7mi/AABnCXex_HInCvRz3kuKLwNna?dl=0

I will pull the GC logs and save them up to the same folder link.
Thank you,
Russ


*Manzama*a MODERN GOVERNANCE company

Russell Bahr
Lead Infrastructure Engineer

USA & CAN Office: +1 (541) 306 3271
USA & CAN Support: +1 (541) 706 9393
UK Office & Support: +44 (0)203 282 1633
AUS Office & Support: +61 (0) 2 8417 2339

543 NW York Drive, Suite 100, Bend, OR 97703

LinkedIn <http://www.linkedin.com/company/manzama> | Twitter
<https://twitter.com/ManzamaInc> | Facebook
<http://www.facebook.com/manzamainc> | YouTube
<https://www.youtube.com/channel/UCBo3QoqewyNoo7HiT_BFuRw>


On Thu, Sep 5, 2019 at 9:38 AM Shawn Heisey <[hidden email]> wrote:

> On 9/4/2019 12:48 PM, Russell Bahr wrote:
> > Thank you for the feedback and advise.  I have loaded the 2 screenshots
> up
> > to drop box.  Here is the link.
> >
> >
> https://www.dropbox.com/s/c5b41a61za0ojw7/solr4_Screen%20Shot%202019-09-03%20at%203.37.08%20PM.png?dl=0
>
> Just one screenshot there.
>
> Looking at that, I'm guessing that you have a heap size between 11 and
> 12 GB, and about 7GB of index data.  A little more than 3GB of your
> system memory is being used for disk caching.  So you can fit about half
> your index into the cache.
>
> I would not ordinarily expect a situation like this to have such bad
> performance, so there may be something going on that isn't
> straightforward to see here.  One possible problem is that your install
> actually needs a bigger heap than you have, so it's spending a lot of
> time doing garbage collection.  If this is what's happening, you
> probably are going to need more than 16GB total memory.
>
> What is the total document count for that 7GB of index data?
>
> Solr 4 likely isn't writing a GC log, but Solr 8 probably is.  Can you
> share those logs?
>
> Thanks,
> Shawn
>
Reply | Threaded
Open this post in threaded view
|

Re: upgrading from solr4 to solr8 searches taking 4 to 10 times as long to return

Russell Bahr
Hi Shawn and Toke,
I have uploaded the solr_gc.log for solr8 and the catalina.out log or solr4
to the link i shared to my dropbox folder.  Did you get a chance to look at
the configs I uploaded?  If you want I can clear out the comments to make
it smaller to read?
Thank you,
Russ

*Manzama*a MODERN GOVERNANCE company

Russell Bahr
Lead Infrastructure Engineer

USA & CAN Office: +1 (541) 306 3271
USA & CAN Support: +1 (541) 706 9393
UK Office & Support: +44 (0)203 282 1633
AUS Office & Support: +61 (0) 2 8417 2339

543 NW York Drive, Suite 100, Bend, OR 97703

LinkedIn <http://www.linkedin.com/company/manzama> | Twitter
<https://twitter.com/ManzamaInc> | Facebook
<http://www.facebook.com/manzamainc> | YouTube
<https://www.youtube.com/channel/UCBo3QoqewyNoo7HiT_BFuRw>


On Thu, Sep 5, 2019 at 9:51 AM Russell Bahr <[hidden email]> wrote:

> Hi Shawn,
> Sorry for the other link.  I figured out after I sent the first one how to
> share the entire folder.  Please try this link and let me know if that
> works.
>
> https://www.dropbox.com/sh/hslknixd3azj7mi/AABnCXex_HInCvRz3kuKLwNna?dl=0
>
> I will pull the GC logs and save them up to the same folder link.
> Thank you,
> Russ
>
>
> *Manzama*a MODERN GOVERNANCE company
>
> Russell Bahr
> Lead Infrastructure Engineer
>
> USA & CAN Office: +1 (541) 306 3271
> USA & CAN Support: +1 (541) 706 9393
> UK Office & Support: +44 (0)203 282 1633
> AUS Office & Support: +61 (0) 2 8417 2339
>
> 543 NW York Drive, Suite 100, Bend, OR 97703
>
> LinkedIn <http://www.linkedin.com/company/manzama> | Twitter
> <https://twitter.com/ManzamaInc> | Facebook
> <http://www.facebook.com/manzamainc> | YouTube
> <https://www.youtube.com/channel/UCBo3QoqewyNoo7HiT_BFuRw>
>
>
> On Thu, Sep 5, 2019 at 9:38 AM Shawn Heisey <[hidden email]> wrote:
>
>> On 9/4/2019 12:48 PM, Russell Bahr wrote:
>> > Thank you for the feedback and advise.  I have loaded the 2 screenshots
>> up
>> > to drop box.  Here is the link.
>> >
>> >
>> https://www.dropbox.com/s/c5b41a61za0ojw7/solr4_Screen%20Shot%202019-09-03%20at%203.37.08%20PM.png?dl=0
>>
>> Just one screenshot there.
>>
>> Looking at that, I'm guessing that you have a heap size between 11 and
>> 12 GB, and about 7GB of index data.  A little more than 3GB of your
>> system memory is being used for disk caching.  So you can fit about half
>> your index into the cache.
>>
>> I would not ordinarily expect a situation like this to have such bad
>> performance, so there may be something going on that isn't
>> straightforward to see here.  One possible problem is that your install
>> actually needs a bigger heap than you have, so it's spending a lot of
>> time doing garbage collection.  If this is what's happening, you
>> probably are going to need more than 16GB total memory.
>>
>> What is the total document count for that 7GB of index data?
>>
>> Solr 4 likely isn't writing a GC log, but Solr 8 probably is.  Can you
>> share those logs?
>>
>> Thanks,
>> Shawn
>>
>
Reply | Threaded
Open this post in threaded view
|

Re: upgrading from solr4 to solr8 searches taking 4 to 10 times as long to return

david.w.smiley@gmail.com
In reply to this post by Russell Bahr
I suggest first working with a single machine to see if it responds
substantially slower with the new version.  Just find one of yours and
issue it a query that will resolve locally (distrib=false param).  Your
current collection level queries are internally issuing such queries, and
so with a little bit of sleuthing, looking at logs, you can find a shard
level query like this.  If it's quick then there's some distributed aspect
to investigate.  But you'll probably see the slowness here, and the problem
is better scoped and easier to diagnose.  At this point look at timings
with debug=timing to see information on each of the components.  That may
give you a strong clue.  If it's in the QueryComponent which actually
executes the underlying search then you have some further digging to do.
Use a profiler like JVisualVM.

~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley
Reply | Threaded
Open this post in threaded view
|

Re: upgrading from solr4 to solr8 searches taking 4 to 10 times as long to return

Russell Bahr
Hi David,
I ran the *:* query 10 times against all 30 servers and the results (below)
were similar across all of them. I agree working against a single server is
easier troubleshooting, but I do not know where to start.

Server shard replica, Matches, Time, Pass
16 1 n2 2989421 78800 1
20 1 n1 2989559 63246 1
23 1 n8 2989619 55141 1
28 1 n6 2989619 65536 1
17 1 n4 2989818 56694 1
26 2 n10 2990088 63485 1
21 2 n18 2990145 68077 1
11 2 n16 2990145 62271 1
13 2 n12 2990242 68564 1
27 2 n14 2990242 63739 1
10 3 n26 2988056 69117 1
25 3 n24 2988056 73750 1
12 3 n28 2988096 61948 1
6 3 n20 2988123 62174 1
19 3 n22 2988123 65826 1
1 4 n30 2985457 60404 1
29 4 n34 2985457 68498 1
30 4 n38 2985604 72034 1
9 4 n36 2902757 65943 1
15 4 n32 2985948 67208 1
7 5 n48 2992278 63098 1
5 5 n42 2992363 69503 1
8 5 n44 2992363 66818 1
4 5 n40 2992397 66784 1
14 5 n46 2883495 58759 1
3 6 n56 2878221 52265 1
22 6 n58 2878221 53768 1
24 6 n52 2878326 62174 1
2 6 n50 2878326 53143 1
18 6 n54 2878326 59044 1

Results from 10 passes
p-solr-8-16.obscured.com:8983/solr/content_shard1_replica_n2/ 69697.8
4599.8171896
Query time milliseconds [78800, 65549, 68045, 72151, 62774, 69168, 66459,
74336, 69028, 70668]
p-solr-8-20.obscured.com:8983/solr/content_shard1_replica_n1/ 58310.5
4531.23621224
Query time milliseconds [63246, 59626, 61001, 59366, 53028, 58693, 58832,
64633, 54659, 50021]
p-solr-8-23.obscured.com:8983/solr/content_shard1_replica_n8/ 57778.5
4659.23933348
Query time milliseconds [55141, 55194, 59100, 62614, 65425, 59261, 58961,
59259, 53799, 49031]
p-solr-8-28.obscured.com:8983/solr/content_shard1_replica_n6/ 64944.1
3382.61379705
Query time milliseconds [65536, 67825, 69829, 60059, 63616, 67588, 68443,
60853, 62666, 63026]
p-solr-8-17.obscured.com:8983/solr/content_shard1_replica_n4/ 58018.9
4821.9028851
Query time milliseconds [56694, 58900, 55404, 51590, 66034, 51256, 57109,
57515, 63530, 62157]
p-solr-8-26.obscured.com:8983/solr/content_shard2_replica_n10/ 59366.6
5036.84751936
Query time milliseconds [63485, 53315, 64845, 62077, 54313, 52607, 65389,
55977, 63486, 58172]
p-solr-8-21.obscured.com:8983/solr/content_shard2_replica_n18/ 61844.1
4623.13444537
Query time milliseconds [68077, 61117, 64284, 65393, 60580, 57495, 58068,
67454, 62370, 53603]
p-solr-8-11.obscured.com:8983/solr/content_shard2_replica_n16/ 61179.1
4224.86040401
Query time milliseconds [62271, 66059, 67076, 55706, 60905, 58617, 56561,
66308, 57100, 61188]
p-solr-8-13.obscured.com:8983/solr/content_shard2_replica_n12/ 69578.3
3986.83530998
Query time milliseconds [68564, 67411, 71644, 75938, 73772, 69780, 67438,
72479, 66368, 62389]
p-solr-8-27.obscured.com:8983/solr/content_shard2_replica_n14/ 59808.2
4896.04649579
Query time milliseconds [63739, 59873, 65775, 50280, 63009, 60955, 55516,
64130, 60016, 54789]
p-solr-8-10.obscured.com:8983/solr/content_shard3_replica_n26/ 66038.1
3363.29340743
Query time milliseconds [69117, 63766, 66189, 72059, 68751, 67644, 65027,
60859, 63306, 63663]
p-solr-8-25.obscured.com:8983/solr/content_shard3_replica_n24/ 60566.8
6390.11408001
Query time milliseconds [73750, 58849, 59947, 58810, 55752, 51790, 54871,
60592, 66853, 64454]
p-solr-8-12.obscured.com:8983/solr/content_shard3_replica_n28/ 63456.6
4591.56887252
Query time milliseconds [61948, 61372, 60570, 59961, 66818, 62124, 66880,
72592, 56477, 65824]
p-solr-8-6.obscured.com:8983/solr/content_shard3_replica_n20/ 60869.6
2619.82752613
Query time milliseconds [62174, 59369, 62098, 66974, 59552, 58757, 59728,
59413, 62414, 58217]
p-solr-8-19.obscured.com:8983/solr/content_shard3_replica_n22/ 57122.0
5111.33222469
Query time milliseconds [65826, 51309, 52595, 50432, 59987, 61371, 62022,
57667, 55639, 54372]
p-solr-8-1.obscured.com:8983/solr/content_shard4_replica_n30/ 61678.3
4091.35454478
Query time milliseconds [60404, 55604, 62858, 67807, 64320, 63962, 66846,
58085, 58951, 57946]
p-solr-8-29.obscured.com:8983/solr/content_shard4_replica_n34/ 64042.9
3646.86466403
Query time milliseconds [68498, 60725, 69115, 61942, 61398, 61874, 61255,
68909, 66059, 60654]
p-solr-8-30.obscured.com:8983/solr/content_shard4_replica_n38/ 63116.5
4609.50824685
Query time milliseconds [72034, 66459, 56206, 56873, 64180, 60061, 63122,
63786, 63754, 64690]
p-solr-8-9.obscured.com:8983/solr/content_shard4_replica_n36/ 63432.7
4300.8959803
Query time milliseconds [65943, 69316, 55002, 68917, 63472, 58904, 63211,
64688, 62819, 62055]
p-solr-8-15.obscured.com:8983/solr/content_shard4_replica_n32/ 61906.5
2902.66395843
Query time milliseconds [67208, 65850, 57653, 60862, 59523, 59620, 62713,
61529, 61265, 62842]
p-solr-8-7.obscured.com:8983/solr/content_shard5_replica_n48/ 59892.7
3490.10076423
Query time milliseconds [63098, 60924, 61086, 63765, 56105, 53150, 56886,
63446, 59949, 60518]
p-solr-8-5.obscured.com:8983/solr/content_shard5_replica_n42/ 62159.0
5601.61476719
Query time milliseconds [69503, 56827, 55902, 61201, 56940, 65581, 71914,
63370, 63051, 57301]
p-solr-8-8.obscured.com:8983/solr/content_shard5_replica_n44/ 63823.7
3858.87292267
Query time milliseconds [66818, 67892, 58788, 69172, 64827, 63549, 63268,
59215, 58640, 66068]
p-solr-8-4.obscured.com:8983/solr/content_shard5_replica_n40/ 58128.5
4130.89318429
Query time milliseconds [66784, 56183, 61574, 55183, 57776, 57367, 60974,
52964, 53819, 58661]
p-solr-8-14.obscured.com:8983/solr/content_shard5_replica_n46/ 61351.6
4037.90460511
Query time milliseconds [58759, 62324, 57665, 67046, 56590, 59296, 59116,
68687, 63815, 60218]
p-solr-8-3.obscured.com:8983/solr/content_shard6_replica_n56/ 56639.2
4509.2680559
Query time milliseconds [52265, 49244, 51974, 60207, 57325, 58683, 63489,
54787, 61009, 57409]
p-solr-8-22.obscured.com:8983/solr/content_shard6_replica_n58/ 57532.8
2690.59624949
Query time milliseconds [53768, 57966, 57145, 53659, 60496, 56692, 59881,
56240, 61887, 57594]
p-solr-8-24.obscured.com:8983/solr/content_shard6_replica_n52/ 65245.2
3561.3285536
Query time milliseconds [62174, 63386, 59160, 68495, 70150, 64167, 70341,
66024, 64306, 64249]
p-solr-8-2.obscured.com:8983/solr/content_shard6_replica_n50/ 58632.8
3571.57904077
Query time milliseconds [53143, 56298, 58081, 62260, 61433, 61427, 63434,
59773, 56407, 54072]
p-solr-8-18.obscured.com:8983/solr/content_shard6_replica_n54/ 62589.2
4958.34174341
Query time milliseconds [59044, 64196, 68406, 63014, 61170, 59517, 63885,
72458, 57918, 56284]

Thank you,
Russ

*Manzama*a MODERN GOVERNANCE company

Russell Bahr
Lead Infrastructure Engineer

USA & CAN Office: +1 (541) 306 3271
USA & CAN Support: +1 (541) 706 9393
UK Office & Support: +44 (0)203 282 1633
AUS Office & Support: +61 (0) 2 8417 2339

543 NW York Drive, Suite 100, Bend, OR 97703

LinkedIn <http://www.linkedin.com/company/manzama> | Twitter
<https://twitter.com/ManzamaInc> | Facebook
<http://www.facebook.com/manzamainc> | YouTube
<https://www.youtube.com/channel/UCBo3QoqewyNoo7HiT_BFuRw>


On Thu, Sep 5, 2019 at 8:50 PM David Smiley <[hidden email]>
wrote:

> I suggest first working with a single machine to see if it responds
> substantially slower with the new version.  Just find one of yours and
> issue it a query that will resolve locally (distrib=false param).  Your
> current collection level queries are internally issuing such queries, and
> so with a little bit of sleuthing, looking at logs, you can find a shard
> level query like this.  If it's quick then there's some distributed aspect
> to investigate.  But you'll probably see the slowness here, and the problem
> is better scoped and easier to diagnose.  At this point look at timings
> with debug=timing to see information on each of the components.  That may
> give you a strong clue.  If it's in the QueryComponent which actually
> executes the underlying search then you have some further digging to do.
> Use a profiler like JVisualVM.
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
Reply | Threaded
Open this post in threaded view
|

Re: upgrading from solr4 to solr8 searches taking 4 to 10 times as long to return

Russell Bahr
In reply to this post by Russell Bahr
Hi Shawn and Toke,
Did you get a chance to look at the configs and logs that I posted?
Thank you again for your help,
Russ

*Manzama*a MODERN GOVERNANCE company

Russell Bahr
Lead Infrastructure Engineer

USA & CAN Office: +1 (541) 306 3271
USA & CAN Support: +1 (541) 706 9393
UK Office & Support: +44 (0)203 282 1633
AUS Office & Support: +61 (0) 2 8417 2339

543 NW York Drive, Suite 100, Bend, OR 97703

LinkedIn <http://www.linkedin.com/company/manzama> | Twitter
<https://twitter.com/ManzamaInc> | Facebook
<http://www.facebook.com/manzamainc> | YouTube
<https://www.youtube.com/channel/UCBo3QoqewyNoo7HiT_BFuRw>


On Thu, Sep 5, 2019 at 5:18 PM Russell Bahr <[hidden email]> wrote:

> Hi Shawn and Toke,
> I have uploaded the solr_gc.log for solr8 and the catalina.out log or
> solr4 to the link i shared to my dropbox folder.  Did you get a chance to
> look at the configs I uploaded?  If you want I can clear out the comments
> to make it smaller to read?
> Thank you,
> Russ
>
> *Manzama*a MODERN GOVERNANCE company
>
> Russell Bahr
> Lead Infrastructure Engineer
>
> USA & CAN Office: +1 (541) 306 3271
> USA & CAN Support: +1 (541) 706 9393
> UK Office & Support: +44 (0)203 282 1633
> AUS Office & Support: +61 (0) 2 8417 2339
>
> 543 NW York Drive, Suite 100, Bend, OR 97703
>
> LinkedIn <http://www.linkedin.com/company/manzama> | Twitter
> <https://twitter.com/ManzamaInc> | Facebook
> <http://www.facebook.com/manzamainc> | YouTube
> <https://www.youtube.com/channel/UCBo3QoqewyNoo7HiT_BFuRw>
>
>
> On Thu, Sep 5, 2019 at 9:51 AM Russell Bahr <[hidden email]> wrote:
>
>> Hi Shawn,
>> Sorry for the other link.  I figured out after I sent the first one how
>> to share the entire folder.  Please try this link and let me know if that
>> works.
>>
>> https://www.dropbox.com/sh/hslknixd3azj7mi/AABnCXex_HInCvRz3kuKLwNna?dl=0
>>
>> I will pull the GC logs and save them up to the same folder link.
>> Thank you,
>> Russ
>>
>>
>> *Manzama*a MODERN GOVERNANCE company
>>
>> Russell Bahr
>> Lead Infrastructure Engineer
>>
>> USA & CAN Office: +1 (541) 306 3271
>> USA & CAN Support: +1 (541) 706 9393
>> UK Office & Support: +44 (0)203 282 1633
>> AUS Office & Support: +61 (0) 2 8417 2339
>>
>> 543 NW York Drive, Suite 100, Bend, OR 97703
>>
>> LinkedIn <http://www.linkedin.com/company/manzama> | Twitter
>> <https://twitter.com/ManzamaInc> | Facebook
>> <http://www.facebook.com/manzamainc> | YouTube
>> <https://www.youtube.com/channel/UCBo3QoqewyNoo7HiT_BFuRw>
>>
>>
>> On Thu, Sep 5, 2019 at 9:38 AM Shawn Heisey <[hidden email]> wrote:
>>
>>> On 9/4/2019 12:48 PM, Russell Bahr wrote:
>>> > Thank you for the feedback and advise.  I have loaded the 2
>>> screenshots up
>>> > to drop box.  Here is the link.
>>> >
>>> >
>>> https://www.dropbox.com/s/c5b41a61za0ojw7/solr4_Screen%20Shot%202019-09-03%20at%203.37.08%20PM.png?dl=0
>>>
>>> Just one screenshot there.
>>>
>>> Looking at that, I'm guessing that you have a heap size between 11 and
>>> 12 GB, and about 7GB of index data.  A little more than 3GB of your
>>> system memory is being used for disk caching.  So you can fit about half
>>> your index into the cache.
>>>
>>> I would not ordinarily expect a situation like this to have such bad
>>> performance, so there may be something going on that isn't
>>> straightforward to see here.  One possible problem is that your install
>>> actually needs a bigger heap than you have, so it's spending a lot of
>>> time doing garbage collection.  If this is what's happening, you
>>> probably are going to need more than 16GB total memory.
>>>
>>> What is the total document count for that 7GB of index data?
>>>
>>> Solr 4 likely isn't writing a GC log, but Solr 8 probably is.  Can you
>>> share those logs?
>>>
>>> Thanks,
>>> Shawn
>>>
>>
Reply | Threaded
Open this post in threaded view
|

Re: upgrading from solr4 to solr8 searches taking 4 to 10 times as long to return

david.w.smiley@gmail.com
In reply to this post by Russell Bahr
10s of seconds to respond to a simple match-all query, especially to just a
single shard via using distrib=false, is very bizarre.  What's the "QTime"
on one of these? -- also super long or sub-second?

I took a brief look at your schema with a hunch.  I see you have
docValues=true on your ID field -- makes sense to me.  You also have
version=1.5 on the schema instead of 1.6.  Why did you not do 1.6?  With
1.5 useDocValuesAsStored is false by default.  try toggling the version
number to 1.6.  And try your query with "fl=id" and see how that changes
the times.

I also took a look at your solrconfig.xml with a hunch, and now think I
found the smoking gun.  I see you've modified the /select request handler
to add a bunch of defaults, including, of all things, grouping.  Thus when
you report to us your queries are simple *:* queries, the reality is far
different.  I wish people would treat /select as immutable and instead
create request handlers for their apps' needs.

Nonetheless my investigation here only reveals that your test queries are
actually very complex and thus explains their overall slowness.  We don't
know why Solr 8 performs slower than Solr 4 here.  For that I think we've
given you some tips.  Get back to a simple query and compare that.  Try
certain features in isolation (e.g. *just* the grouping).  Maybe it's
that.  You might experiment with switching "fingerprint" (the string field
you group on) from docValues=true to false to see if it's a docValues perf
issue compared to uninverting.

~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley


On Sat, Sep 7, 2019 at 3:06 PM Russell Bahr <[hidden email]> wrote:

> Hi David,
> I ran the *:* query 10 times against all 30 servers and the results (below)
> were similar across all of them. I agree working against a single server is
> easier troubleshooting, but I do not know where to start.
>
> Server shard replica, Matches, Time, Pass
> 16 1 n2 2989421 78800 1
> 20 1 n1 2989559 63246 1
> 23 1 n8 2989619 55141 1
> 28 1 n6 2989619 65536 1
> 17 1 n4 2989818 56694 1
> 26 2 n10 2990088 63485 1
> 21 2 n18 2990145 68077 1
> 11 2 n16 2990145 62271 1
> 13 2 n12 2990242 68564 1
> 27 2 n14 2990242 63739 1
> 10 3 n26 2988056 69117 1
> 25 3 n24 2988056 73750 1
> 12 3 n28 2988096 61948 1
> 6 3 n20 2988123 62174 1
> 19 3 n22 2988123 65826 1
> 1 4 n30 2985457 60404 1
> 29 4 n34 2985457 68498 1
> 30 4 n38 2985604 72034 1
> 9 4 n36 2902757 65943 1
> 15 4 n32 2985948 67208 1
> 7 5 n48 2992278 63098 1
> 5 5 n42 2992363 69503 1
> 8 5 n44 2992363 66818 1
> 4 5 n40 2992397 66784 1
> 14 5 n46 2883495 58759 1
> 3 6 n56 2878221 52265 1
> 22 6 n58 2878221 53768 1
> 24 6 n52 2878326 62174 1
> 2 6 n50 2878326 53143 1
> 18 6 n54 2878326 59044 1
>
> Results from 10 passes
> p-solr-8-16.obscured.com:8983/solr/content_shard1_replica_n2/ 69697.8
> 4599.8171896
> Query time milliseconds [78800, 65549, 68045, 72151, 62774, 69168, 66459,
> 74336, 69028, 70668]
> p-solr-8-20.obscured.com:8983/solr/content_shard1_replica_n1/ 58310.5
> 4531.23621224
> Query time milliseconds [63246, 59626, 61001, 59366, 53028, 58693, 58832,
> 64633, 54659, 50021]
> p-solr-8-23.obscured.com:8983/solr/content_shard1_replica_n8/ 57778.5
> 4659.23933348
> Query time milliseconds [55141, 55194, 59100, 62614, 65425, 59261, 58961,
> 59259, 53799, 49031]
> p-solr-8-28.obscured.com:8983/solr/content_shard1_replica_n6/ 64944.1
> 3382.61379705
> Query time milliseconds [65536, 67825, 69829, 60059, 63616, 67588, 68443,
> 60853, 62666, 63026]
> p-solr-8-17.obscured.com:8983/solr/content_shard1_replica_n4/ 58018.9
> 4821.9028851
> Query time milliseconds [56694, 58900, 55404, 51590, 66034, 51256, 57109,
> 57515, 63530, 62157]
> p-solr-8-26.obscured.com:8983/solr/content_shard2_replica_n10/ 59366.6
> 5036.84751936
> Query time milliseconds [63485, 53315, 64845, 62077, 54313, 52607, 65389,
> 55977, 63486, 58172]
> p-solr-8-21.obscured.com:8983/solr/content_shard2_replica_n18/ 61844.1
> 4623.13444537
> Query time milliseconds [68077, 61117, 64284, 65393, 60580, 57495, 58068,
> 67454, 62370, 53603]
> p-solr-8-11.obscured.com:8983/solr/content_shard2_replica_n16/ 61179.1
> 4224.86040401
> Query time milliseconds [62271, 66059, 67076, 55706, 60905, 58617, 56561,
> 66308, 57100, 61188]
> p-solr-8-13.obscured.com:8983/solr/content_shard2_replica_n12/ 69578.3
> 3986.83530998
> Query time milliseconds [68564, 67411, 71644, 75938, 73772, 69780, 67438,
> 72479, 66368, 62389]
> p-solr-8-27.obscured.com:8983/solr/content_shard2_replica_n14/ 59808.2
> 4896.04649579
> Query time milliseconds [63739, 59873, 65775, 50280, 63009, 60955, 55516,
> 64130, 60016, 54789]
> p-solr-8-10.obscured.com:8983/solr/content_shard3_replica_n26/ 66038.1
> 3363.29340743
> Query time milliseconds [69117, 63766, 66189, 72059, 68751, 67644, 65027,
> 60859, 63306, 63663]
> p-solr-8-25.obscured.com:8983/solr/content_shard3_replica_n24/ 60566.8
> 6390.11408001
> Query time milliseconds [73750, 58849, 59947, 58810, 55752, 51790, 54871,
> 60592, 66853, 64454]
> p-solr-8-12.obscured.com:8983/solr/content_shard3_replica_n28/ 63456.6
> 4591.56887252
> Query time milliseconds [61948, 61372, 60570, 59961, 66818, 62124, 66880,
> 72592, 56477, 65824]
> p-solr-8-6.obscured.com:8983/solr/content_shard3_replica_n20/ 60869.6
> 2619.82752613
> Query time milliseconds [62174, 59369, 62098, 66974, 59552, 58757, 59728,
> 59413, 62414, 58217]
> p-solr-8-19.obscured.com:8983/solr/content_shard3_replica_n22/ 57122.0
> 5111.33222469
> Query time milliseconds [65826, 51309, 52595, 50432, 59987, 61371, 62022,
> 57667, 55639, 54372]
> p-solr-8-1.obscured.com:8983/solr/content_shard4_replica_n30/ 61678.3
> 4091.35454478
> Query time milliseconds [60404, 55604, 62858, 67807, 64320, 63962, 66846,
> 58085, 58951, 57946]
> p-solr-8-29.obscured.com:8983/solr/content_shard4_replica_n34/ 64042.9
> 3646.86466403
> Query time milliseconds [68498, 60725, 69115, 61942, 61398, 61874, 61255,
> 68909, 66059, 60654]
> p-solr-8-30.obscured.com:8983/solr/content_shard4_replica_n38/ 63116.5
> 4609.50824685
> Query time milliseconds [72034, 66459, 56206, 56873, 64180, 60061, 63122,
> 63786, 63754, 64690]
> p-solr-8-9.obscured.com:8983/solr/content_shard4_replica_n36/ 63432.7
> 4300.8959803
> Query time milliseconds [65943, 69316, 55002, 68917, 63472, 58904, 63211,
> 64688, 62819, 62055]
> p-solr-8-15.obscured.com:8983/solr/content_shard4_replica_n32/ 61906.5
> 2902.66395843
> Query time milliseconds [67208, 65850, 57653, 60862, 59523, 59620, 62713,
> 61529, 61265, 62842]
> p-solr-8-7.obscured.com:8983/solr/content_shard5_replica_n48/ 59892.7
> 3490.10076423
> Query time milliseconds [63098, 60924, 61086, 63765, 56105, 53150, 56886,
> 63446, 59949, 60518]
> p-solr-8-5.obscured.com:8983/solr/content_shard5_replica_n42/ 62159.0
> 5601.61476719
> Query time milliseconds [69503, 56827, 55902, 61201, 56940, 65581, 71914,
> 63370, 63051, 57301]
> p-solr-8-8.obscured.com:8983/solr/content_shard5_replica_n44/ 63823.7
> 3858.87292267
> Query time milliseconds [66818, 67892, 58788, 69172, 64827, 63549, 63268,
> 59215, 58640, 66068]
> p-solr-8-4.obscured.com:8983/solr/content_shard5_replica_n40/ 58128.5
> 4130.89318429
> Query time milliseconds [66784, 56183, 61574, 55183, 57776, 57367, 60974,
> 52964, 53819, 58661]
> p-solr-8-14.obscured.com:8983/solr/content_shard5_replica_n46/ 61351.6
> 4037.90460511
> Query time milliseconds [58759, 62324, 57665, 67046, 56590, 59296, 59116,
> 68687, 63815, 60218]
> p-solr-8-3.obscured.com:8983/solr/content_shard6_replica_n56/ 56639.2
> 4509.2680559
> Query time milliseconds [52265, 49244, 51974, 60207, 57325, 58683, 63489,
> 54787, 61009, 57409]
> p-solr-8-22.obscured.com:8983/solr/content_shard6_replica_n58/ 57532.8
> 2690.59624949
> Query time milliseconds [53768, 57966, 57145, 53659, 60496, 56692, 59881,
> 56240, 61887, 57594]
> p-solr-8-24.obscured.com:8983/solr/content_shard6_replica_n52/ 65245.2
> 3561.3285536
> Query time milliseconds [62174, 63386, 59160, 68495, 70150, 64167, 70341,
> 66024, 64306, 64249]
> p-solr-8-2.obscured.com:8983/solr/content_shard6_replica_n50/ 58632.8
> 3571.57904077
> Query time milliseconds [53143, 56298, 58081, 62260, 61433, 61427, 63434,
> 59773, 56407, 54072]
> p-solr-8-18.obscured.com:8983/solr/content_shard6_replica_n54/ 62589.2
> 4958.34174341
> Query time milliseconds [59044, 64196, 68406, 63014, 61170, 59517, 63885,
> 72458, 57918, 56284]
>
> Thank you,
> Russ
>
> *Manzama*a MODERN GOVERNANCE company
>
> Russell Bahr
> Lead Infrastructure Engineer
>
> USA & CAN Office: +1 (541) 306 3271
> USA & CAN Support: +1 (541) 706 9393
> UK Office & Support: +44 (0)203 282 1633
> AUS Office & Support: +61 (0) 2 8417 2339
>
> 543 NW York Drive, Suite 100, Bend, OR 97703
>
> LinkedIn <http://www.linkedin.com/company/manzama> | Twitter
> <https://twitter.com/ManzamaInc> | Facebook
> <http://www.facebook.com/manzamainc> | YouTube
> <https://www.youtube.com/channel/UCBo3QoqewyNoo7HiT_BFuRw>
>
>
> On Thu, Sep 5, 2019 at 8:50 PM David Smiley <[hidden email]>
> wrote:
>
> > I suggest first working with a single machine to see if it responds
> > substantially slower with the new version.  Just find one of yours and
> > issue it a query that will resolve locally (distrib=false param).  Your
> > current collection level queries are internally issuing such queries, and
> > so with a little bit of sleuthing, looking at logs, you can find a shard
> > level query like this.  If it's quick then there's some distributed
> aspect
> > to investigate.  But you'll probably see the slowness here, and the
> problem
> > is better scoped and easier to diagnose.  At this point look at timings
> > with debug=timing to see information on each of the components.  That may
> > give you a strong clue.  If it's in the QueryComponent which actually
> > executes the underlying search then you have some further digging to do.
> > Use a profiler like JVisualVM.
> >
> > ~ David Smiley
> > Apache Lucene/Solr Search Developer
> > http://www.linkedin.com/in/davidwsmiley
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: upgrading from solr4 to solr8 searches taking 4 to 10 times as long to return

Toke Eskildsen-2
In reply to this post by Russell Bahr
Russell Bahr <[hidden email]> wrote:
> Did you get a chance to look at the configs and logs that I posted?

Sorry, firefighting at work and the weekend happened. Don't really have time now either, but I took a look in your solrconfig.xml after David noticed the group-thing.

Grouping in itself isn't necessarily that bad, but ngroups can be quite heavy (and is tricky with distributed searches, so I hope you read the documentation on that). Try setting group.ngroups=false (doing so in your request should override what you have in solrconfig.xml) and if that does not change anything, try disabling grouping fully.

It does not explain the difference between Solr 4 & 8, but I agree with David that we need to isolate what causes the overall slowdown first, before we can attempt to fix the Solr 4 vs 8 thing.

- Toke Eskildsen
Reply | Threaded
Open this post in threaded view
|

Re: upgrading from solr4 to solr8 searches taking 4 to 10 times as long to return

david.w.smiley@gmail.com
In reply to this post by david.w.smiley@gmail.com
Also consider substituting grouping with expand/collapse (see the ref
guide).  The latter performs much better in general, although grouping does
have certain options that are uniquely valuable like ensuring that facet
counts look at the aggregate (if you want that).  I wish we could outright
remove grouping; it's a complexity weight on our codebase.

~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley


On Sat, Sep 7, 2019 at 5:15 PM David Smiley <[hidden email]>
wrote:

> 10s of seconds to respond to a simple match-all query, especially to just
> a single shard via using distrib=false, is very bizarre.  What's the
> "QTime" on one of these? -- also super long or sub-second?
>
> I took a brief look at your schema with a hunch.  I see you have
> docValues=true on your ID field -- makes sense to me.  You also have
> version=1.5 on the schema instead of 1.6.  Why did you not do 1.6?  With
> 1.5 useDocValuesAsStored is false by default.  try toggling the version
> number to 1.6.  And try your query with "fl=id" and see how that changes
> the times.
>
> I also took a look at your solrconfig.xml with a hunch, and now think I
> found the smoking gun.  I see you've modified the /select request handler
> to add a bunch of defaults, including, of all things, grouping.  Thus when
> you report to us your queries are simple *:* queries, the reality is far
> different.  I wish people would treat /select as immutable and instead
> create request handlers for their apps' needs.
>
> Nonetheless my investigation here only reveals that your test queries are
> actually very complex and thus explains their overall slowness.  We don't
> know why Solr 8 performs slower than Solr 4 here.  For that I think we've
> given you some tips.  Get back to a simple query and compare that.  Try
> certain features in isolation (e.g. *just* the grouping).  Maybe it's
> that.  You might experiment with switching "fingerprint" (the string field
> you group on) from docValues=true to false to see if it's a docValues perf
> issue compared to uninverting.
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
>
> On Sat, Sep 7, 2019 at 3:06 PM Russell Bahr <[hidden email]> wrote:
>
>> Hi David,
>> I ran the *:* query 10 times against all 30 servers and the results
>> (below)
>> were similar across all of them. I agree working against a single server
>> is
>> easier troubleshooting, but I do not know where to start.
>>
>> Server shard replica, Matches, Time, Pass
>> 16 1 n2 2989421 78800 1
>> 20 1 n1 2989559 63246 1
>> 23 1 n8 2989619 55141 1
>> 28 1 n6 2989619 65536 1
>> 17 1 n4 2989818 56694 1
>> 26 2 n10 2990088 63485 1
>> 21 2 n18 2990145 68077 1
>> 11 2 n16 2990145 62271 1
>> 13 2 n12 2990242 68564 1
>> 27 2 n14 2990242 63739 1
>> 10 3 n26 2988056 69117 1
>> 25 3 n24 2988056 73750 1
>> 12 3 n28 2988096 61948 1
>> 6 3 n20 2988123 62174 1
>> 19 3 n22 2988123 65826 1
>> 1 4 n30 2985457 60404 1
>> 29 4 n34 2985457 68498 1
>> 30 4 n38 2985604 72034 1
>> 9 4 n36 2902757 65943 1
>> 15 4 n32 2985948 67208 1
>> 7 5 n48 2992278 63098 1
>> 5 5 n42 2992363 69503 1
>> 8 5 n44 2992363 66818 1
>> 4 5 n40 2992397 66784 1
>> 14 5 n46 2883495 58759 1
>> 3 6 n56 2878221 52265 1
>> 22 6 n58 2878221 53768 1
>> 24 6 n52 2878326 62174 1
>> 2 6 n50 2878326 53143 1
>> 18 6 n54 2878326 59044 1
>>
>> Results from 10 passes
>> p-solr-8-16.obscured.com:8983/solr/content_shard1_replica_n2/ 69697.8
>> 4599.8171896
>> Query time milliseconds [78800, 65549, 68045, 72151, 62774, 69168, 66459,
>> 74336, 69028, 70668]
>> p-solr-8-20.obscured.com:8983/solr/content_shard1_replica_n1/ 58310.5
>> 4531.23621224
>> Query time milliseconds [63246, 59626, 61001, 59366, 53028, 58693, 58832,
>> 64633, 54659, 50021]
>> p-solr-8-23.obscured.com:8983/solr/content_shard1_replica_n8/ 57778.5
>> 4659.23933348
>> Query time milliseconds [55141, 55194, 59100, 62614, 65425, 59261, 58961,
>> 59259, 53799, 49031]
>> p-solr-8-28.obscured.com:8983/solr/content_shard1_replica_n6/ 64944.1
>> 3382.61379705
>> Query time milliseconds [65536, 67825, 69829, 60059, 63616, 67588, 68443,
>> 60853, 62666, 63026]
>> p-solr-8-17.obscured.com:8983/solr/content_shard1_replica_n4/ 58018.9
>> 4821.9028851
>> Query time milliseconds [56694, 58900, 55404, 51590, 66034, 51256, 57109,
>> 57515, 63530, 62157]
>> p-solr-8-26.obscured.com:8983/solr/content_shard2_replica_n10/ 59366.6
>> 5036.84751936
>> Query time milliseconds [63485, 53315, 64845, 62077, 54313, 52607, 65389,
>> 55977, 63486, 58172]
>> p-solr-8-21.obscured.com:8983/solr/content_shard2_replica_n18/ 61844.1
>> 4623.13444537
>> Query time milliseconds [68077, 61117, 64284, 65393, 60580, 57495, 58068,
>> 67454, 62370, 53603]
>> p-solr-8-11.obscured.com:8983/solr/content_shard2_replica_n16/ 61179.1
>> 4224.86040401
>> Query time milliseconds [62271, 66059, 67076, 55706, 60905, 58617, 56561,
>> 66308, 57100, 61188]
>> p-solr-8-13.obscured.com:8983/solr/content_shard2_replica_n12/ 69578.3
>> 3986.83530998
>> Query time milliseconds [68564, 67411, 71644, 75938, 73772, 69780, 67438,
>> 72479, 66368, 62389]
>> p-solr-8-27.obscured.com:8983/solr/content_shard2_replica_n14/ 59808.2
>> 4896.04649579
>> Query time milliseconds [63739, 59873, 65775, 50280, 63009, 60955, 55516,
>> 64130, 60016, 54789]
>> p-solr-8-10.obscured.com:8983/solr/content_shard3_replica_n26/ 66038.1
>> 3363.29340743
>> Query time milliseconds [69117, 63766, 66189, 72059, 68751, 67644, 65027,
>> 60859, 63306, 63663]
>> p-solr-8-25.obscured.com:8983/solr/content_shard3_replica_n24/ 60566.8
>> 6390.11408001
>> Query time milliseconds [73750, 58849, 59947, 58810, 55752, 51790, 54871,
>> 60592, 66853, 64454]
>> p-solr-8-12.obscured.com:8983/solr/content_shard3_replica_n28/ 63456.6
>> 4591.56887252
>> Query time milliseconds [61948, 61372, 60570, 59961, 66818, 62124, 66880,
>> 72592, 56477, 65824]
>> p-solr-8-6.obscured.com:8983/solr/content_shard3_replica_n20/ 60869.6
>> 2619.82752613
>> Query time milliseconds [62174, 59369, 62098, 66974, 59552, 58757, 59728,
>> 59413, 62414, 58217]
>> p-solr-8-19.obscured.com:8983/solr/content_shard3_replica_n22/ 57122.0
>> 5111.33222469
>> Query time milliseconds [65826, 51309, 52595, 50432, 59987, 61371, 62022,
>> 57667, 55639, 54372]
>> p-solr-8-1.obscured.com:8983/solr/content_shard4_replica_n30/ 61678.3
>> 4091.35454478
>> Query time milliseconds [60404, 55604, 62858, 67807, 64320, 63962, 66846,
>> 58085, 58951, 57946]
>> p-solr-8-29.obscured.com:8983/solr/content_shard4_replica_n34/ 64042.9
>> 3646.86466403
>> Query time milliseconds [68498, 60725, 69115, 61942, 61398, 61874, 61255,
>> 68909, 66059, 60654]
>> p-solr-8-30.obscured.com:8983/solr/content_shard4_replica_n38/ 63116.5
>> 4609.50824685
>> Query time milliseconds [72034, 66459, 56206, 56873, 64180, 60061, 63122,
>> 63786, 63754, 64690]
>> p-solr-8-9.obscured.com:8983/solr/content_shard4_replica_n36/ 63432.7
>> 4300.8959803
>> Query time milliseconds [65943, 69316, 55002, 68917, 63472, 58904, 63211,
>> 64688, 62819, 62055]
>> p-solr-8-15.obscured.com:8983/solr/content_shard4_replica_n32/ 61906.5
>> 2902.66395843
>> Query time milliseconds [67208, 65850, 57653, 60862, 59523, 59620, 62713,
>> 61529, 61265, 62842]
>> p-solr-8-7.obscured.com:8983/solr/content_shard5_replica_n48/ 59892.7
>> 3490.10076423
>> Query time milliseconds [63098, 60924, 61086, 63765, 56105, 53150, 56886,
>> 63446, 59949, 60518]
>> p-solr-8-5.obscured.com:8983/solr/content_shard5_replica_n42/ 62159.0
>> 5601.61476719
>> Query time milliseconds [69503, 56827, 55902, 61201, 56940, 65581, 71914,
>> 63370, 63051, 57301]
>> p-solr-8-8.obscured.com:8983/solr/content_shard5_replica_n44/ 63823.7
>> 3858.87292267
>> Query time milliseconds [66818, 67892, 58788, 69172, 64827, 63549, 63268,
>> 59215, 58640, 66068]
>> p-solr-8-4.obscured.com:8983/solr/content_shard5_replica_n40/ 58128.5
>> 4130.89318429
>> Query time milliseconds [66784, 56183, 61574, 55183, 57776, 57367, 60974,
>> 52964, 53819, 58661]
>> p-solr-8-14.obscured.com:8983/solr/content_shard5_replica_n46/ 61351.6
>> 4037.90460511
>> Query time milliseconds [58759, 62324, 57665, 67046, 56590, 59296, 59116,
>> 68687, 63815, 60218]
>> p-solr-8-3.obscured.com:8983/solr/content_shard6_replica_n56/ 56639.2
>> 4509.2680559
>> Query time milliseconds [52265, 49244, 51974, 60207, 57325, 58683, 63489,
>> 54787, 61009, 57409]
>> p-solr-8-22.obscured.com:8983/solr/content_shard6_replica_n58/ 57532.8
>> 2690.59624949
>> Query time milliseconds [53768, 57966, 57145, 53659, 60496, 56692, 59881,
>> 56240, 61887, 57594]
>> p-solr-8-24.obscured.com:8983/solr/content_shard6_replica_n52/ 65245.2
>> 3561.3285536
>> Query time milliseconds [62174, 63386, 59160, 68495, 70150, 64167, 70341,
>> 66024, 64306, 64249]
>> p-solr-8-2.obscured.com:8983/solr/content_shard6_replica_n50/ 58632.8
>> 3571.57904077
>> Query time milliseconds [53143, 56298, 58081, 62260, 61433, 61427, 63434,
>> 59773, 56407, 54072]
>> p-solr-8-18.obscured.com:8983/solr/content_shard6_replica_n54/ 62589.2
>> 4958.34174341
>> Query time milliseconds [59044, 64196, 68406, 63014, 61170, 59517, 63885,
>> 72458, 57918, 56284]
>>
>> Thank you,
>> Russ
>>
>> *Manzama*a MODERN GOVERNANCE company
>>
>> Russell Bahr
>> Lead Infrastructure Engineer
>>
>> USA & CAN Office: +1 (541) 306 3271
>> USA & CAN Support: +1 (541) 706 9393
>> UK Office & Support: +44 (0)203 282 1633
>> AUS Office & Support: +61 (0) 2 8417 2339
>>
>> 543 NW York Drive, Suite 100, Bend, OR 97703
>>
>> LinkedIn <http://www.linkedin.com/company/manzama> | Twitter
>> <https://twitter.com/ManzamaInc> | Facebook
>> <http://www.facebook.com/manzamainc> | YouTube
>> <https://www.youtube.com/channel/UCBo3QoqewyNoo7HiT_BFuRw>
>>
>>
>> On Thu, Sep 5, 2019 at 8:50 PM David Smiley <[hidden email]>
>> wrote:
>>
>> > I suggest first working with a single machine to see if it responds
>> > substantially slower with the new version.  Just find one of yours and
>> > issue it a query that will resolve locally (distrib=false param).  Your
>> > current collection level queries are internally issuing such queries,
>> and
>> > so with a little bit of sleuthing, looking at logs, you can find a shard
>> > level query like this.  If it's quick then there's some distributed
>> aspect
>> > to investigate.  But you'll probably see the slowness here, and the
>> problem
>> > is better scoped and easier to diagnose.  At this point look at timings
>> > with debug=timing to see information on each of the components.  That
>> may
>> > give you a strong clue.  If it's in the QueryComponent which actually
>> > executes the underlying search then you have some further digging to do.
>> > Use a profiler like JVisualVM.
>> >
>> > ~ David Smiley
>> > Apache Lucene/Solr Search Developer
>> > http://www.linkedin.com/in/davidwsmiley
>> >
>>
>
12