upgrading from solr4 to solr8 searches taking 4 to 10 times as long to return

classic Classic list List threaded Threaded
26 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Re: upgrading from solr4 to solr8 searches taking 4 to 10 times as long to return

Russell Bahr
Hi David and Toke,
Thank you both for your input.  I will be in DC tomorrow evening and will
try your suggestions, and read the ref guide again on the parts that you
have pointed out.  I will let you know the results, and will share your
feedback with my team to see what we can change and still bring back the
result sets that are needed for our system.
Thanks again,
Russ

*Manzama*a MODERN GOVERNANCE company

Russell Bahr
Lead Infrastructure Engineer

USA & CAN Office: +1 (541) 306 3271
USA & CAN Support: +1 (541) 706 9393
UK Office & Support: +44 (0)203 282 1633
AUS Office & Support: +61 (0) 2 8417 2339

543 NW York Drive, Suite 100, Bend, OR 97703

LinkedIn <http://www.linkedin.com/company/manzama> | Twitter
<https://twitter.com/ManzamaInc> | Facebook
<http://www.facebook.com/manzamainc> | YouTube
<https://www.youtube.com/channel/UCBo3QoqewyNoo7HiT_BFuRw>


On Sat, Sep 7, 2019 at 2:43 PM David Smiley <[hidden email]>
wrote:

> Also consider substituting grouping with expand/collapse (see the ref
> guide).  The latter performs much better in general, although grouping does
> have certain options that are uniquely valuable like ensuring that facet
> counts look at the aggregate (if you want that).  I wish we could outright
> remove grouping; it's a complexity weight on our codebase.
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
>
> On Sat, Sep 7, 2019 at 5:15 PM David Smiley <[hidden email]>
> wrote:
>
> > 10s of seconds to respond to a simple match-all query, especially to just
> > a single shard via using distrib=false, is very bizarre.  What's the
> > "QTime" on one of these? -- also super long or sub-second?
> >
> > I took a brief look at your schema with a hunch.  I see you have
> > docValues=true on your ID field -- makes sense to me.  You also have
> > version=1.5 on the schema instead of 1.6.  Why did you not do 1.6?  With
> > 1.5 useDocValuesAsStored is false by default.  try toggling the version
> > number to 1.6.  And try your query with "fl=id" and see how that changes
> > the times.
> >
> > I also took a look at your solrconfig.xml with a hunch, and now think I
> > found the smoking gun.  I see you've modified the /select request handler
> > to add a bunch of defaults, including, of all things, grouping.  Thus
> when
> > you report to us your queries are simple *:* queries, the reality is far
> > different.  I wish people would treat /select as immutable and instead
> > create request handlers for their apps' needs.
> >
> > Nonetheless my investigation here only reveals that your test queries are
> > actually very complex and thus explains their overall slowness.  We don't
> > know why Solr 8 performs slower than Solr 4 here.  For that I think we've
> > given you some tips.  Get back to a simple query and compare that.  Try
> > certain features in isolation (e.g. *just* the grouping).  Maybe it's
> > that.  You might experiment with switching "fingerprint" (the string
> field
> > you group on) from docValues=true to false to see if it's a docValues
> perf
> > issue compared to uninverting.
> >
> > ~ David Smiley
> > Apache Lucene/Solr Search Developer
> > http://www.linkedin.com/in/davidwsmiley
> >
> >
> > On Sat, Sep 7, 2019 at 3:06 PM Russell Bahr <[hidden email]> wrote:
> >
> >> Hi David,
> >> I ran the *:* query 10 times against all 30 servers and the results
> >> (below)
> >> were similar across all of them. I agree working against a single server
> >> is
> >> easier troubleshooting, but I do not know where to start.
> >>
> >> Server shard replica, Matches, Time, Pass
> >> 16 1 n2 2989421 78800 1
> >> 20 1 n1 2989559 63246 1
> >> 23 1 n8 2989619 55141 1
> >> 28 1 n6 2989619 65536 1
> >> 17 1 n4 2989818 56694 1
> >> 26 2 n10 2990088 63485 1
> >> 21 2 n18 2990145 68077 1
> >> 11 2 n16 2990145 62271 1
> >> 13 2 n12 2990242 68564 1
> >> 27 2 n14 2990242 63739 1
> >> 10 3 n26 2988056 69117 1
> >> 25 3 n24 2988056 73750 1
> >> 12 3 n28 2988096 61948 1
> >> 6 3 n20 2988123 62174 1
> >> 19 3 n22 2988123 65826 1
> >> 1 4 n30 2985457 60404 1
> >> 29 4 n34 2985457 68498 1
> >> 30 4 n38 2985604 72034 1
> >> 9 4 n36 2902757 65943 1
> >> 15 4 n32 2985948 67208 1
> >> 7 5 n48 2992278 63098 1
> >> 5 5 n42 2992363 69503 1
> >> 8 5 n44 2992363 66818 1
> >> 4 5 n40 2992397 66784 1
> >> 14 5 n46 2883495 58759 1
> >> 3 6 n56 2878221 52265 1
> >> 22 6 n58 2878221 53768 1
> >> 24 6 n52 2878326 62174 1
> >> 2 6 n50 2878326 53143 1
> >> 18 6 n54 2878326 59044 1
> >>
> >> Results from 10 passes
> >> p-solr-8-16.obscured.com:8983/solr/content_shard1_replica_n2/ 69697.8
> >> 4599.8171896
> >> Query time milliseconds [78800, 65549, 68045, 72151, 62774, 69168,
> 66459,
> >> 74336, 69028, 70668]
> >> p-solr-8-20.obscured.com:8983/solr/content_shard1_replica_n1/ 58310.5
> >> 4531.23621224
> >> Query time milliseconds [63246, 59626, 61001, 59366, 53028, 58693,
> 58832,
> >> 64633, 54659, 50021]
> >> p-solr-8-23.obscured.com:8983/solr/content_shard1_replica_n8/ 57778.5
> >> 4659.23933348
> >> Query time milliseconds [55141, 55194, 59100, 62614, 65425, 59261,
> 58961,
> >> 59259, 53799, 49031]
> >> p-solr-8-28.obscured.com:8983/solr/content_shard1_replica_n6/ 64944.1
> >> 3382.61379705
> >> Query time milliseconds [65536, 67825, 69829, 60059, 63616, 67588,
> 68443,
> >> 60853, 62666, 63026]
> >> p-solr-8-17.obscured.com:8983/solr/content_shard1_replica_n4/ 58018.9
> >> 4821.9028851
> >> Query time milliseconds [56694, 58900, 55404, 51590, 66034, 51256,
> 57109,
> >> 57515, 63530, 62157]
> >> p-solr-8-26.obscured.com:8983/solr/content_shard2_replica_n10/ 59366.6
> >> 5036.84751936
> >> Query time milliseconds [63485, 53315, 64845, 62077, 54313, 52607,
> 65389,
> >> 55977, 63486, 58172]
> >> p-solr-8-21.obscured.com:8983/solr/content_shard2_replica_n18/ 61844.1
> >> 4623.13444537
> >> Query time milliseconds [68077, 61117, 64284, 65393, 60580, 57495,
> 58068,
> >> 67454, 62370, 53603]
> >> p-solr-8-11.obscured.com:8983/solr/content_shard2_replica_n16/ 61179.1
> >> 4224.86040401
> >> Query time milliseconds [62271, 66059, 67076, 55706, 60905, 58617,
> 56561,
> >> 66308, 57100, 61188]
> >> p-solr-8-13.obscured.com:8983/solr/content_shard2_replica_n12/ 69578.3
> >> 3986.83530998
> >> Query time milliseconds [68564, 67411, 71644, 75938, 73772, 69780,
> 67438,
> >> 72479, 66368, 62389]
> >> p-solr-8-27.obscured.com:8983/solr/content_shard2_replica_n14/ 59808.2
> >> 4896.04649579
> >> Query time milliseconds [63739, 59873, 65775, 50280, 63009, 60955,
> 55516,
> >> 64130, 60016, 54789]
> >> p-solr-8-10.obscured.com:8983/solr/content_shard3_replica_n26/ 66038.1
> >> 3363.29340743
> >> Query time milliseconds [69117, 63766, 66189, 72059, 68751, 67644,
> 65027,
> >> 60859, 63306, 63663]
> >> p-solr-8-25.obscured.com:8983/solr/content_shard3_replica_n24/ 60566.8
> >> 6390.11408001
> >> Query time milliseconds [73750, 58849, 59947, 58810, 55752, 51790,
> 54871,
> >> 60592, 66853, 64454]
> >> p-solr-8-12.obscured.com:8983/solr/content_shard3_replica_n28/ 63456.6
> >> 4591.56887252
> >> Query time milliseconds [61948, 61372, 60570, 59961, 66818, 62124,
> 66880,
> >> 72592, 56477, 65824]
> >> p-solr-8-6.obscured.com:8983/solr/content_shard3_replica_n20/ 60869.6
> >> 2619.82752613
> >> Query time milliseconds [62174, 59369, 62098, 66974, 59552, 58757,
> 59728,
> >> 59413, 62414, 58217]
> >> p-solr-8-19.obscured.com:8983/solr/content_shard3_replica_n22/ 57122.0
> >> 5111.33222469
> >> Query time milliseconds [65826, 51309, 52595, 50432, 59987, 61371,
> 62022,
> >> 57667, 55639, 54372]
> >> p-solr-8-1.obscured.com:8983/solr/content_shard4_replica_n30/ 61678.3
> >> 4091.35454478
> >> Query time milliseconds [60404, 55604, 62858, 67807, 64320, 63962,
> 66846,
> >> 58085, 58951, 57946]
> >> p-solr-8-29.obscured.com:8983/solr/content_shard4_replica_n34/ 64042.9
> >> 3646.86466403
> >> Query time milliseconds [68498, 60725, 69115, 61942, 61398, 61874,
> 61255,
> >> 68909, 66059, 60654]
> >> p-solr-8-30.obscured.com:8983/solr/content_shard4_replica_n38/ 63116.5
> >> 4609.50824685
> >> Query time milliseconds [72034, 66459, 56206, 56873, 64180, 60061,
> 63122,
> >> 63786, 63754, 64690]
> >> p-solr-8-9.obscured.com:8983/solr/content_shard4_replica_n36/ 63432.7
> >> 4300.8959803
> >> Query time milliseconds [65943, 69316, 55002, 68917, 63472, 58904,
> 63211,
> >> 64688, 62819, 62055]
> >> p-solr-8-15.obscured.com:8983/solr/content_shard4_replica_n32/ 61906.5
> >> 2902.66395843
> >> Query time milliseconds [67208, 65850, 57653, 60862, 59523, 59620,
> 62713,
> >> 61529, 61265, 62842]
> >> p-solr-8-7.obscured.com:8983/solr/content_shard5_replica_n48/ 59892.7
> >> 3490.10076423
> >> Query time milliseconds [63098, 60924, 61086, 63765, 56105, 53150,
> 56886,
> >> 63446, 59949, 60518]
> >> p-solr-8-5.obscured.com:8983/solr/content_shard5_replica_n42/ 62159.0
> >> 5601.61476719
> >> Query time milliseconds [69503, 56827, 55902, 61201, 56940, 65581,
> 71914,
> >> 63370, 63051, 57301]
> >> p-solr-8-8.obscured.com:8983/solr/content_shard5_replica_n44/ 63823.7
> >> 3858.87292267
> >> Query time milliseconds [66818, 67892, 58788, 69172, 64827, 63549,
> 63268,
> >> 59215, 58640, 66068]
> >> p-solr-8-4.obscured.com:8983/solr/content_shard5_replica_n40/ 58128.5
> >> 4130.89318429
> >> Query time milliseconds [66784, 56183, 61574, 55183, 57776, 57367,
> 60974,
> >> 52964, 53819, 58661]
> >> p-solr-8-14.obscured.com:8983/solr/content_shard5_replica_n46/ 61351.6
> >> 4037.90460511
> >> Query time milliseconds [58759, 62324, 57665, 67046, 56590, 59296,
> 59116,
> >> 68687, 63815, 60218]
> >> p-solr-8-3.obscured.com:8983/solr/content_shard6_replica_n56/ 56639.2
> >> 4509.2680559
> >> Query time milliseconds [52265, 49244, 51974, 60207, 57325, 58683,
> 63489,
> >> 54787, 61009, 57409]
> >> p-solr-8-22.obscured.com:8983/solr/content_shard6_replica_n58/ 57532.8
> >> 2690.59624949
> >> Query time milliseconds [53768, 57966, 57145, 53659, 60496, 56692,
> 59881,
> >> 56240, 61887, 57594]
> >> p-solr-8-24.obscured.com:8983/solr/content_shard6_replica_n52/ 65245.2
> >> 3561.3285536
> >> Query time milliseconds [62174, 63386, 59160, 68495, 70150, 64167,
> 70341,
> >> 66024, 64306, 64249]
> >> p-solr-8-2.obscured.com:8983/solr/content_shard6_replica_n50/ 58632.8
> >> 3571.57904077
> >> Query time milliseconds [53143, 56298, 58081, 62260, 61433, 61427,
> 63434,
> >> 59773, 56407, 54072]
> >> p-solr-8-18.obscured.com:8983/solr/content_shard6_replica_n54/ 62589.2
> >> 4958.34174341
> >> Query time milliseconds [59044, 64196, 68406, 63014, 61170, 59517,
> 63885,
> >> 72458, 57918, 56284]
> >>
> >> Thank you,
> >> Russ
> >>
> >> *Manzama*a MODERN GOVERNANCE company
> >>
> >> Russell Bahr
> >> Lead Infrastructure Engineer
> >>
> >> USA & CAN Office: +1 (541) 306 3271
> >> USA & CAN Support: +1 (541) 706 9393
> >> UK Office & Support: +44 (0)203 282 1633
> >> AUS Office & Support: +61 (0) 2 8417 2339
> >>
> >> 543 NW York Drive, Suite 100, Bend, OR 97703
> >>
> >> LinkedIn <http://www.linkedin.com/company/manzama> | Twitter
> >> <https://twitter.com/ManzamaInc> | Facebook
> >> <http://www.facebook.com/manzamainc> | YouTube
> >> <https://www.youtube.com/channel/UCBo3QoqewyNoo7HiT_BFuRw>
> >>
> >>
> >> On Thu, Sep 5, 2019 at 8:50 PM David Smiley <[hidden email]>
> >> wrote:
> >>
> >> > I suggest first working with a single machine to see if it responds
> >> > substantially slower with the new version.  Just find one of yours and
> >> > issue it a query that will resolve locally (distrib=false param).
> Your
> >> > current collection level queries are internally issuing such queries,
> >> and
> >> > so with a little bit of sleuthing, looking at logs, you can find a
> shard
> >> > level query like this.  If it's quick then there's some distributed
> >> aspect
> >> > to investigate.  But you'll probably see the slowness here, and the
> >> problem
> >> > is better scoped and easier to diagnose.  At this point look at
> timings
> >> > with debug=timing to see information on each of the components.  That
> >> may
> >> > give you a strong clue.  If it's in the QueryComponent which actually
> >> > executes the underlying search then you have some further digging to
> do.
> >> > Use a profiler like JVisualVM.
> >> >
> >> > ~ David Smiley
> >> > Apache Lucene/Solr Search Developer
> >> > http://www.linkedin.com/in/davidwsmiley
> >> >
> >>
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: upgrading from solr4 to solr8 searches taking 4 to 10 times as long to return

guenterh.lists@bluewin.ch
Hi

what about this
https://issues.apache.org/jira/browse/SOLR-8096
seems still unresolved issue?

With our migration from version 4 to 7 last year we experienced similar
problems.

Günter

On 08.09.19 06:09, Russell Bahr wrote:

> Hi David and Toke,
> Thank you both for your input.  I will be in DC tomorrow evening and will
> try your suggestions, and read the ref guide again on the parts that you
> have pointed out.  I will let you know the results, and will share your
> feedback with my team to see what we can change and still bring back the
> result sets that are needed for our system.
> Thanks again,
> Russ
>
> *Manzama*a MODERN GOVERNANCE company
>
> Russell Bahr
> Lead Infrastructure Engineer
>
> USA & CAN Office: +1 (541) 306 3271
> USA & CAN Support: +1 (541) 706 9393
> UK Office & Support: +44 (0)203 282 1633
> AUS Office & Support: +61 (0) 2 8417 2339
>
> 543 NW York Drive, Suite 100, Bend, OR 97703
>
> LinkedIn <http://www.linkedin.com/company/manzama> | Twitter
> <https://twitter.com/ManzamaInc> | Facebook
> <http://www.facebook.com/manzamainc> | YouTube
> <https://www.youtube.com/channel/UCBo3QoqewyNoo7HiT_BFuRw>
>
>
> On Sat, Sep 7, 2019 at 2:43 PM David Smiley <[hidden email]>
> wrote:
>
>> Also consider substituting grouping with expand/collapse (see the ref
>> guide).  The latter performs much better in general, although grouping does
>> have certain options that are uniquely valuable like ensuring that facet
>> counts look at the aggregate (if you want that).  I wish we could outright
>> remove grouping; it's a complexity weight on our codebase.
>>
>> ~ David Smiley
>> Apache Lucene/Solr Search Developer
>> http://www.linkedin.com/in/davidwsmiley
>>
>>
>> On Sat, Sep 7, 2019 at 5:15 PM David Smiley <[hidden email]>
>> wrote:
>>
>>> 10s of seconds to respond to a simple match-all query, especially to just
>>> a single shard via using distrib=false, is very bizarre.  What's the
>>> "QTime" on one of these? -- also super long or sub-second?
>>>
>>> I took a brief look at your schema with a hunch.  I see you have
>>> docValues=true on your ID field -- makes sense to me.  You also have
>>> version=1.5 on the schema instead of 1.6.  Why did you not do 1.6?  With
>>> 1.5 useDocValuesAsStored is false by default.  try toggling the version
>>> number to 1.6.  And try your query with "fl=id" and see how that changes
>>> the times.
>>>
>>> I also took a look at your solrconfig.xml with a hunch, and now think I
>>> found the smoking gun.  I see you've modified the /select request handler
>>> to add a bunch of defaults, including, of all things, grouping.  Thus
>> when
>>> you report to us your queries are simple *:* queries, the reality is far
>>> different.  I wish people would treat /select as immutable and instead
>>> create request handlers for their apps' needs.
>>>
>>> Nonetheless my investigation here only reveals that your test queries are
>>> actually very complex and thus explains their overall slowness.  We don't
>>> know why Solr 8 performs slower than Solr 4 here.  For that I think we've
>>> given you some tips.  Get back to a simple query and compare that.  Try
>>> certain features in isolation (e.g. *just* the grouping).  Maybe it's
>>> that.  You might experiment with switching "fingerprint" (the string
>> field
>>> you group on) from docValues=true to false to see if it's a docValues
>> perf
>>> issue compared to uninverting.
>>>
>>> ~ David Smiley
>>> Apache Lucene/Solr Search Developer
>>> http://www.linkedin.com/in/davidwsmiley
>>>
>>>
>>> On Sat, Sep 7, 2019 at 3:06 PM Russell Bahr <[hidden email]> wrote:
>>>
>>>> Hi David,
>>>> I ran the *:* query 10 times against all 30 servers and the results
>>>> (below)
>>>> were similar across all of them. I agree working against a single server
>>>> is
>>>> easier troubleshooting, but I do not know where to start.
>>>>
>>>> Server shard replica, Matches, Time, Pass
>>>> 16 1 n2 2989421 78800 1
>>>> 20 1 n1 2989559 63246 1
>>>> 23 1 n8 2989619 55141 1
>>>> 28 1 n6 2989619 65536 1
>>>> 17 1 n4 2989818 56694 1
>>>> 26 2 n10 2990088 63485 1
>>>> 21 2 n18 2990145 68077 1
>>>> 11 2 n16 2990145 62271 1
>>>> 13 2 n12 2990242 68564 1
>>>> 27 2 n14 2990242 63739 1
>>>> 10 3 n26 2988056 69117 1
>>>> 25 3 n24 2988056 73750 1
>>>> 12 3 n28 2988096 61948 1
>>>> 6 3 n20 2988123 62174 1
>>>> 19 3 n22 2988123 65826 1
>>>> 1 4 n30 2985457 60404 1
>>>> 29 4 n34 2985457 68498 1
>>>> 30 4 n38 2985604 72034 1
>>>> 9 4 n36 2902757 65943 1
>>>> 15 4 n32 2985948 67208 1
>>>> 7 5 n48 2992278 63098 1
>>>> 5 5 n42 2992363 69503 1
>>>> 8 5 n44 2992363 66818 1
>>>> 4 5 n40 2992397 66784 1
>>>> 14 5 n46 2883495 58759 1
>>>> 3 6 n56 2878221 52265 1
>>>> 22 6 n58 2878221 53768 1
>>>> 24 6 n52 2878326 62174 1
>>>> 2 6 n50 2878326 53143 1
>>>> 18 6 n54 2878326 59044 1
>>>>
>>>> Results from 10 passes
>>>> p-solr-8-16.obscured.com:8983/solr/content_shard1_replica_n2/ 69697.8
>>>> 4599.8171896
>>>> Query time milliseconds [78800, 65549, 68045, 72151, 62774, 69168,
>> 66459,
>>>> 74336, 69028, 70668]
>>>> p-solr-8-20.obscured.com:8983/solr/content_shard1_replica_n1/ 58310.5
>>>> 4531.23621224
>>>> Query time milliseconds [63246, 59626, 61001, 59366, 53028, 58693,
>> 58832,
>>>> 64633, 54659, 50021]
>>>> p-solr-8-23.obscured.com:8983/solr/content_shard1_replica_n8/ 57778.5
>>>> 4659.23933348
>>>> Query time milliseconds [55141, 55194, 59100, 62614, 65425, 59261,
>> 58961,
>>>> 59259, 53799, 49031]
>>>> p-solr-8-28.obscured.com:8983/solr/content_shard1_replica_n6/ 64944.1
>>>> 3382.61379705
>>>> Query time milliseconds [65536, 67825, 69829, 60059, 63616, 67588,
>> 68443,
>>>> 60853, 62666, 63026]
>>>> p-solr-8-17.obscured.com:8983/solr/content_shard1_replica_n4/ 58018.9
>>>> 4821.9028851
>>>> Query time milliseconds [56694, 58900, 55404, 51590, 66034, 51256,
>> 57109,
>>>> 57515, 63530, 62157]
>>>> p-solr-8-26.obscured.com:8983/solr/content_shard2_replica_n10/ 59366.6
>>>> 5036.84751936
>>>> Query time milliseconds [63485, 53315, 64845, 62077, 54313, 52607,
>> 65389,
>>>> 55977, 63486, 58172]
>>>> p-solr-8-21.obscured.com:8983/solr/content_shard2_replica_n18/ 61844.1
>>>> 4623.13444537
>>>> Query time milliseconds [68077, 61117, 64284, 65393, 60580, 57495,
>> 58068,
>>>> 67454, 62370, 53603]
>>>> p-solr-8-11.obscured.com:8983/solr/content_shard2_replica_n16/ 61179.1
>>>> 4224.86040401
>>>> Query time milliseconds [62271, 66059, 67076, 55706, 60905, 58617,
>> 56561,
>>>> 66308, 57100, 61188]
>>>> p-solr-8-13.obscured.com:8983/solr/content_shard2_replica_n12/ 69578.3
>>>> 3986.83530998
>>>> Query time milliseconds [68564, 67411, 71644, 75938, 73772, 69780,
>> 67438,
>>>> 72479, 66368, 62389]
>>>> p-solr-8-27.obscured.com:8983/solr/content_shard2_replica_n14/ 59808.2
>>>> 4896.04649579
>>>> Query time milliseconds [63739, 59873, 65775, 50280, 63009, 60955,
>> 55516,
>>>> 64130, 60016, 54789]
>>>> p-solr-8-10.obscured.com:8983/solr/content_shard3_replica_n26/ 66038.1
>>>> 3363.29340743
>>>> Query time milliseconds [69117, 63766, 66189, 72059, 68751, 67644,
>> 65027,
>>>> 60859, 63306, 63663]
>>>> p-solr-8-25.obscured.com:8983/solr/content_shard3_replica_n24/ 60566.8
>>>> 6390.11408001
>>>> Query time milliseconds [73750, 58849, 59947, 58810, 55752, 51790,
>> 54871,
>>>> 60592, 66853, 64454]
>>>> p-solr-8-12.obscured.com:8983/solr/content_shard3_replica_n28/ 63456.6
>>>> 4591.56887252
>>>> Query time milliseconds [61948, 61372, 60570, 59961, 66818, 62124,
>> 66880,
>>>> 72592, 56477, 65824]
>>>> p-solr-8-6.obscured.com:8983/solr/content_shard3_replica_n20/ 60869.6
>>>> 2619.82752613
>>>> Query time milliseconds [62174, 59369, 62098, 66974, 59552, 58757,
>> 59728,
>>>> 59413, 62414, 58217]
>>>> p-solr-8-19.obscured.com:8983/solr/content_shard3_replica_n22/ 57122.0
>>>> 5111.33222469
>>>> Query time milliseconds [65826, 51309, 52595, 50432, 59987, 61371,
>> 62022,
>>>> 57667, 55639, 54372]
>>>> p-solr-8-1.obscured.com:8983/solr/content_shard4_replica_n30/ 61678.3
>>>> 4091.35454478
>>>> Query time milliseconds [60404, 55604, 62858, 67807, 64320, 63962,
>> 66846,
>>>> 58085, 58951, 57946]
>>>> p-solr-8-29.obscured.com:8983/solr/content_shard4_replica_n34/ 64042.9
>>>> 3646.86466403
>>>> Query time milliseconds [68498, 60725, 69115, 61942, 61398, 61874,
>> 61255,
>>>> 68909, 66059, 60654]
>>>> p-solr-8-30.obscured.com:8983/solr/content_shard4_replica_n38/ 63116.5
>>>> 4609.50824685
>>>> Query time milliseconds [72034, 66459, 56206, 56873, 64180, 60061,
>> 63122,
>>>> 63786, 63754, 64690]
>>>> p-solr-8-9.obscured.com:8983/solr/content_shard4_replica_n36/ 63432.7
>>>> 4300.8959803
>>>> Query time milliseconds [65943, 69316, 55002, 68917, 63472, 58904,
>> 63211,
>>>> 64688, 62819, 62055]
>>>> p-solr-8-15.obscured.com:8983/solr/content_shard4_replica_n32/ 61906.5
>>>> 2902.66395843
>>>> Query time milliseconds [67208, 65850, 57653, 60862, 59523, 59620,
>> 62713,
>>>> 61529, 61265, 62842]
>>>> p-solr-8-7.obscured.com:8983/solr/content_shard5_replica_n48/ 59892.7
>>>> 3490.10076423
>>>> Query time milliseconds [63098, 60924, 61086, 63765, 56105, 53150,
>> 56886,
>>>> 63446, 59949, 60518]
>>>> p-solr-8-5.obscured.com:8983/solr/content_shard5_replica_n42/ 62159.0
>>>> 5601.61476719
>>>> Query time milliseconds [69503, 56827, 55902, 61201, 56940, 65581,
>> 71914,
>>>> 63370, 63051, 57301]
>>>> p-solr-8-8.obscured.com:8983/solr/content_shard5_replica_n44/ 63823.7
>>>> 3858.87292267
>>>> Query time milliseconds [66818, 67892, 58788, 69172, 64827, 63549,
>> 63268,
>>>> 59215, 58640, 66068]
>>>> p-solr-8-4.obscured.com:8983/solr/content_shard5_replica_n40/ 58128.5
>>>> 4130.89318429
>>>> Query time milliseconds [66784, 56183, 61574, 55183, 57776, 57367,
>> 60974,
>>>> 52964, 53819, 58661]
>>>> p-solr-8-14.obscured.com:8983/solr/content_shard5_replica_n46/ 61351.6
>>>> 4037.90460511
>>>> Query time milliseconds [58759, 62324, 57665, 67046, 56590, 59296,
>> 59116,
>>>> 68687, 63815, 60218]
>>>> p-solr-8-3.obscured.com:8983/solr/content_shard6_replica_n56/ 56639.2
>>>> 4509.2680559
>>>> Query time milliseconds [52265, 49244, 51974, 60207, 57325, 58683,
>> 63489,
>>>> 54787, 61009, 57409]
>>>> p-solr-8-22.obscured.com:8983/solr/content_shard6_replica_n58/ 57532.8
>>>> 2690.59624949
>>>> Query time milliseconds [53768, 57966, 57145, 53659, 60496, 56692,
>> 59881,
>>>> 56240, 61887, 57594]
>>>> p-solr-8-24.obscured.com:8983/solr/content_shard6_replica_n52/ 65245.2
>>>> 3561.3285536
>>>> Query time milliseconds [62174, 63386, 59160, 68495, 70150, 64167,
>> 70341,
>>>> 66024, 64306, 64249]
>>>> p-solr-8-2.obscured.com:8983/solr/content_shard6_replica_n50/ 58632.8
>>>> 3571.57904077
>>>> Query time milliseconds [53143, 56298, 58081, 62260, 61433, 61427,
>> 63434,
>>>> 59773, 56407, 54072]
>>>> p-solr-8-18.obscured.com:8983/solr/content_shard6_replica_n54/ 62589.2
>>>> 4958.34174341
>>>> Query time milliseconds [59044, 64196, 68406, 63014, 61170, 59517,
>> 63885,
>>>> 72458, 57918, 56284]
>>>>
>>>> Thank you,
>>>> Russ
>>>>
>>>> *Manzama*a MODERN GOVERNANCE company
>>>>
>>>> Russell Bahr
>>>> Lead Infrastructure Engineer
>>>>
>>>> USA & CAN Office: +1 (541) 306 3271
>>>> USA & CAN Support: +1 (541) 706 9393
>>>> UK Office & Support: +44 (0)203 282 1633
>>>> AUS Office & Support: +61 (0) 2 8417 2339
>>>>
>>>> 543 NW York Drive, Suite 100, Bend, OR 97703
>>>>
>>>> LinkedIn <http://www.linkedin.com/company/manzama> | Twitter
>>>> <https://twitter.com/ManzamaInc> | Facebook
>>>> <http://www.facebook.com/manzamainc> | YouTube
>>>> <https://www.youtube.com/channel/UCBo3QoqewyNoo7HiT_BFuRw>
>>>>
>>>>
>>>> On Thu, Sep 5, 2019 at 8:50 PM David Smiley <[hidden email]>
>>>> wrote:
>>>>
>>>>> I suggest first working with a single machine to see if it responds
>>>>> substantially slower with the new version.  Just find one of yours and
>>>>> issue it a query that will resolve locally (distrib=false param).
>> Your
>>>>> current collection level queries are internally issuing such queries,
>>>> and
>>>>> so with a little bit of sleuthing, looking at logs, you can find a
>> shard
>>>>> level query like this.  If it's quick then there's some distributed
>>>> aspect
>>>>> to investigate.  But you'll probably see the slowness here, and the
>>>> problem
>>>>> is better scoped and easier to diagnose.  At this point look at
>> timings
>>>>> with debug=timing to see information on each of the components.  That
>>>> may
>>>>> give you a strong clue.  If it's in the QueryComponent which actually
>>>>> executes the underlying search then you have some further digging to
>> do.
>>>>> Use a profiler like JVisualVM.
>>>>>
>>>>> ~ David Smiley
>>>>> Apache Lucene/Solr Search Developer
>>>>> http://www.linkedin.com/in/davidwsmiley
>>>>>
Reply | Threaded
Open this post in threaded view
|

Re: upgrading from solr4 to solr8 searches taking 4 to 10 times as long to return

Toke Eskildsen-2
Günter Hipler <[hidden email]> wrote:
> what about this
> https://issues.apache.org/jira/browse/SOLR-8096
> seems still unresolved issue?

Unfortunately Russell has de-shared the solrconfig.xml, but as far as I remember it does not trigger faceting.

> With our migration from version 4 to 7 last year we experienced similar
> problems.

The iterator-based DocValues implementation in Solr 7 has a performance issue with large segments, with symptoms akin to SOLR-8096. If you have not already solved your problems, Solr 8 (with an upgraded index) might help.

- Toke Eskildsen
Reply | Threaded
Open this post in threaded view
|

Re: upgrading from solr4 to solr8 searches taking 4 to 10 times as long to return

guenterh.lists@bluewin.ch
Thanks for this information Toke - the library community, your domain
too, is happy to hear this.

I have seen you have done a lot of work at the end of version 7 for
version 8 but was not sure if it is related to this issue.

Best wishes from Basel, Günter

On 08.09.19 19:42, Toke Eskildsen wrote:

> Günter Hipler <[hidden email]> wrote:
>> what about this
>> https://issues.apache.org/jira/browse/SOLR-8096
>> seems still unresolved issue?
> Unfortunately Russell has de-shared the solrconfig.xml, but as far as I remember it does not trigger faceting.
>
>> With our migration from version 4 to 7 last year we experienced similar
>> problems.
> The iterator-based DocValues implementation in Solr 7 has a performance issue with large segments, with symptoms akin to SOLR-8096. If you have not already solved your problems, Solr 8 (with an upgraded index) might help.
>
> - Toke Eskildsen
>
Reply | Threaded
Open this post in threaded view
|

Re: upgrading from solr4 to solr8 searches taking 4 to 10 times as long to return

Toke Eskildsen-2
On Sun, 2019-09-08 at 21:10 +0200, Günter Hipler wrote:
> I have seen you have done a lot of work at the end of version 7 for
> version 8 but was not sure if it is related to this issue.

It is directly related to DocValues, but it is unclear if that is
Russell's challenge. Our setup is at the far end with 300M docs / 900GB
segments. With Solr 7, performance dropped with a factor 10 for
standard searches (as we used populated documents from DocValues
instead of Stored) and a factor 100 for worst-case exports.


Long story short

* Solr 4-6 has random access DocValues
* Solr 7 has iterator based (all previous blocks must be visited to get
to the block containing the needed value)
* Solr 8 has iterator based with skip lists (go directly to the needed
block)


Looking back in the list, I see that you had performance problems with
faceting on Solr 6 too. I'll make a note to read up on that and respond
in that thread, to avoid hi-jacking this one. It probably won't be this
week as Real Work is heating up.

- Toke Eskildsen, Royal Danish Library


Reply | Threaded
Open this post in threaded view
|

Re: upgrading from solr4 to solr8 searches taking 4 to 10 times as long to return

Russell Bahr
In reply to this post by Toke Eskildsen-2
Hi Toke,
I just checked and the link is still active.  I am back from DC today and am going through and continuing to work on the issue.  Here is the link.  Any insights from you on this is appreciated.  I am going back through the previous suggestions and am in the process of trying them now.
Thanks,
Russ

> On Sep 8, 2019, at 10:42 AM, Toke Eskildsen <[hidden email]> wrote:
>
> Günter Hipler <[hidden email]> wrote:
>> what about this
>> https://issues.apache.org/jira/browse/SOLR-8096
>> seems still unresolved issue?
>
> Unfortunately Russell has de-shared the solrconfig.xml, but as far as I remember it does not trigger faceting.
>
>> With our migration from version 4 to 7 last year we experienced similar
>> problems.
>
> The iterator-based DocValues implementation in Solr 7 has a performance issue with large segments, with symptoms akin to SOLR-8096. If you have not already solved your problems, Solr 8 (with an upgraded index) might help.
>
> - Toke Eskildsen

12