Unexpected Performance decrease when upgrading Solr 5.5.2 to 8.5.2

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Unexpected Performance decrease when upgrading Solr 5.5.2 to 8.5.2

Keene Chen

Hi Solr users community,


We have been doing some performance tests on Solr 5.5.2 and Solr 8.5.2 as part of an upgrading process, and we have noticed some reduced performance for certain types of requests, particularly those that requests a large number of rows, eg. 10000. Would anyone have an explanation as to why the performance degrades, and what areas can be looked at in order to improve its performance?


The performance test example below was carried out using 18000 of such queries, running at a constant throughput as specified by the label in the x-axis. “Rpm” here stands for “requests per minute”.


Solr 8.5’s maximum response times are consistently better. However, the 95th and 99th percentile are comparably worse than Solr 5.5’s response times.



The maximum throughput for solr 8.5 is reached sooner than Solr 5.5 at around 4 requests per second.





Regards,
Keene

--


Keene Chen  | Senior Software Developer



Connect with us

LinkedIn  Twitter  Instagram  Facebook  News  Blog




Mintel Group Ltd | 11 Pilgrim Street | London | EC4V 6RN
Registered in England: Number 1475918. | VAT Number: GB 232 9342 72

Contact details for our other offices can be found at http://www.mintel.com/office-locations.

This email and any attachments may include content that is confidential, privileged
or otherwise protected under applicable law. Unauthorised disclosure, copying, distribution
or use of the contents is prohibited and may be unlawful. If you have received this email in error,
including without appropriate authorisation, then please reply to the sender about the error
and delete this email and any attachments.



Mintel Group Ltd | 11 Pilgrim Street | London | EC4V 6RN
Registered in England: Number 1475918. | VAT Number: GB 232 9342 72

Contact details for our other offices can be found at http://www.mintel.com/office-locations.

This email and any attachments may include content that is confidential, privileged
or otherwise protected under applicable law. Unauthorised disclosure, copying, distribution
or use of the contents is prohibited and may be unlawful. If you have received this email in error,
including without appropriate authorisation, then please reply to the sender about the error
and delete this email and any attachments.

Reply | Threaded
Open this post in threaded view
|

Re: Unexpected Performance decrease when upgrading Solr 5.5.2 to 8.5.2

Colvin Cowie
Hello,

Your images won't appear on the mailing list. You'll need to post them
elsewhere and link to them.

On Tue, 15 Sep 2020 at 09:44, Keene Chen <[hidden email]> wrote:

> Hi Solr users community,
>
>
> We have been doing some performance tests on Solr 5.5.2 and Solr 8.5.2 as
> part of an upgrading process, and we have noticed some reduced performance
> for certain types of requests, particularly those that requests a large
> number of rows, eg. 10000. Would anyone have an explanation as to why the
> performance degrades, and what areas can be looked at in order to improve
> its performance?
>
> The performance test example below was carried out using 18000 of such
> queries, running at a constant throughput as specified by the label in the
> x-axis. “Rpm” here stands for “requests per minute”.
>
> Solr 8.5’s maximum response times are consistently better. However, the
> 95th and 99th percentile are comparably worse than Solr 5.5’s response
> times.
> [image: image.png]
> [image: image.png]
>
> The maximum throughput for solr 8.5 is reached sooner than Solr 5.5 at
> around 4 requests per second.
>
>
> [image: image.png]
> [image: image.png]
>
> Regards,
> Keene
>
> --
>
>
> Keene Chen  | Senior Software Developer
>
>
>
> Connect with us
>
> LinkedIn <https://www.linkedin.com/company/mintel/>  Twitter
> <https://twitter.com/mintelnews>  Instagram
> <https://www.instagram.com/mintelnews/>  Facebook
> <https://www.facebook.com/MintelGroup/>  News
> <https://www.mintel.com/press-centre>  Blog <https://www.mintel.com/blog>
>
> <https://www.mintel.com/>
>
>
>
> Mintel Group Ltd | 11 Pilgrim Street | London | EC4V 6RN
> Registered in England: Number 1475918. | VAT Number: GB 232 9342 72
>
> Contact details for our other offices can be found at
> http://www.mintel.com/office-locations.
>
> This email and any attachments may include content that is confidential,
> privileged
> or otherwise protected under applicable law. Unauthorised disclosure,
> copying, distribution
> or use of the contents is prohibited and may be unlawful. If you have
> received this email in error,
> including without appropriate authorisation, then please reply to the
> sender about the error
> and delete this email and any attachments.
>
>
>
> Mintel Group Ltd | 11 Pilgrim Street | London | EC4V 6RN
> Registered in England: Number 1475918. | VAT Number: GB 232 9342 72
>
> Contact details for our other offices can be found at
> http://www.mintel.com/office-locations.
>
> This email and any attachments may include content that is confidential,
> privileged
> or otherwise protected under applicable law. Unauthorised disclosure,
> copying, distribution
> or use of the contents is prohibited and may be unlawful. If you have
> received this email in error,
> including without appropriate authorisation, then please reply to the
> sender about the error
> and delete this email and any attachments.
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Unexpected Performance decrease when upgrading Solr 5.5.2 to 8.5.2

Keene Chen
Hi,

Thanks for pointing that out. I've linked the images below:

solr5_response_times.png
<https://gcdt-solr-tests-public.s3-eu-west-1.amazonaws.com/solr5_response_times.png>

solr8_response_times.png
<https://gcdt-solr-tests-public.s3-eu-west-1.amazonaws.com/solr8_response_times.png>

solr5_throughput.png
<https://gcdt-solr-tests-public.s3-eu-west-1.amazonaws.com/solr5_throughput.png>

solr8_throughput.png
<https://gcdt-solr-tests-public.s3-eu-west-1.amazonaws.com/solr8_throughput.png>

Regards,
Keene


On Wed, 16 Sep 2020 at 09:09, Colvin Cowie <[hidden email]>
wrote:

> Hello,
>
> Your images won't appear on the mailing list. You'll need to post them
> elsewhere and link to them.
>
> On Tue, 15 Sep 2020 at 09:44, Keene Chen <[hidden email]> wrote:
>
> > Hi Solr users community,
> >
> >
> > We have been doing some performance tests on Solr 5.5.2 and Solr 8.5.2 as
> > part of an upgrading process, and we have noticed some reduced
> performance
> > for certain types of requests, particularly those that requests a large
> > number of rows, eg. 10000. Would anyone have an explanation as to why the
> > performance degrades, and what areas can be looked at in order to improve
> > its performance?
> >
> > The performance test example below was carried out using 18000 of such
> > queries, running at a constant throughput as specified by the label in
> the
> > x-axis. “Rpm” here stands for “requests per minute”.
> >
> > Solr 8.5’s maximum response times are consistently better. However, the
> > 95th and 99th percentile are comparably worse than Solr 5.5’s response
> > times.
> > [image: image.png]
> > [image: image.png]
> >
> > The maximum throughput for solr 8.5 is reached sooner than Solr 5.5 at
> > around 4 requests per second.
> >
> >
> > [image: image.png]
> > [image: image.png]
> >
> > Regards,
> > Keene
> >
> > --
> >
> >
> > Keene Chen  | Senior Software Developer
> >
> >
> >
> > Connect with us
> >
> > LinkedIn <https://www.linkedin.com/company/mintel/>  Twitter
> > <https://twitter.com/mintelnews>  Instagram
> > <https://www.instagram.com/mintelnews/>  Facebook
> > <https://www.facebook.com/MintelGroup/>  News
> > <https://www.mintel.com/press-centre>  Blog <https://www.mintel.com/blog
> >
> >
> > <https://www.mintel.com/>
> >
> >
> >
> > Mintel Group Ltd | 11 Pilgrim Street | London | EC4V 6RN
> > Registered in England: Number 1475918. | VAT Number: GB 232 9342 72
> >
> > Contact details for our other offices can be found at
> > http://www.mintel.com/office-locations.
> >
> > This email and any attachments may include content that is confidential,
> > privileged
> > or otherwise protected under applicable law. Unauthorised disclosure,
> > copying, distribution
> > or use of the contents is prohibited and may be unlawful. If you have
> > received this email in error,
> > including without appropriate authorisation, then please reply to the
> > sender about the error
> > and delete this email and any attachments.
> >
> >
> >
> > Mintel Group Ltd | 11 Pilgrim Street | London | EC4V 6RN
> > Registered in England: Number 1475918. | VAT Number: GB 232 9342 72
> >
> > Contact details for our other offices can be found at
> > http://www.mintel.com/office-locations.
> >
> > This email and any attachments may include content that is confidential,
> > privileged
> > or otherwise protected under applicable law. Unauthorised disclosure,
> > copying, distribution
> > or use of the contents is prohibited and may be unlawful. If you have
> > received this email in error,
> > including without appropriate authorisation, then please reply to the
> > sender about the error
> > and delete this email and any attachments.
> >
> >
>

--

Mintel Group Ltd | 11 Pilgrim Street | London | EC4V 6RN
Registered in
England: Number 1475918. | VAT Number: GB 232 9342 72

Contact details for
our other offices can be found at http://www.mintel.com/office-locations 
<http://www.mintel.com/office-locations>.

This email and any attachments
may include content that is confidential, privileged
or otherwise
protected under applicable law. Unauthorised disclosure, copying,
distribution
or use of the contents is prohibited and may be unlawful. If
you have received this email in error,
including without appropriate
authorisation, then please reply to the sender about the error
and delete
this email and any attachments.

Reply | Threaded
Open this post in threaded view
|

Re: Unexpected Performance decrease when upgrading Solr 5.5.2 to 8.5.2

Toke Eskildsen-2
In reply to this post by Keene Chen
Keene Chen <[hidden email]> wrote:
> We have been doing some performance tests on Solr 5.5.2
> and Solr 8.5.2 as part of an upgrading process, and we have
> noticed some reduced performance for certain types of
> requests, particularly those that requests a large number of
> rows, eg. 10000.

Solr 5→8… One large change is the switch to streaming docvalues in Solr 7, which has an effect on random access speed, especially if there are many documents per shard and only some of the documents has a value for the field. This should have been improved in Solr 8, so a factor 10 slowdown (median, 60 users) is surprising to me.

Could you tell us how many fields you return for each document and how many of these are marked as Stored or/and DocValues? Also approximately how many documents you have per shard?

And a sanity check: The default behaviour of document retrieval with regard to DocValues changed with schema version 1.6 (https://issues.apache.org/jira/browse/SOLR-8220). Have you checked that the same number of fields are returned for the two setups?

- Toke Eskildsen?
Reply | Threaded
Open this post in threaded view
|

Re: Unexpected Performance decrease when upgrading Solr 5.5.2 to 8.5.2

matthew sporleder
In reply to this post by Keene Chen
Did you re-work your schema at all?  There are new primitive types,
new lucene versions, DocValue's, etc

On Wed, Sep 16, 2020 at 12:40 PM Keene Chen <[hidden email]> wrote:

>
> Hi,
>
> Thanks for pointing that out. I've linked the images below:
>
> solr5_response_times.png
> <https://gcdt-solr-tests-public.s3-eu-west-1.amazonaws.com/solr5_response_times.png>
>
> solr8_response_times.png
> <https://gcdt-solr-tests-public.s3-eu-west-1.amazonaws.com/solr8_response_times.png>
>
> solr5_throughput.png
> <https://gcdt-solr-tests-public.s3-eu-west-1.amazonaws.com/solr5_throughput.png>
>
> solr8_throughput.png
> <https://gcdt-solr-tests-public.s3-eu-west-1.amazonaws.com/solr8_throughput.png>
>
> Regards,
> Keene
>
>
> On Wed, 16 Sep 2020 at 09:09, Colvin Cowie <[hidden email]>
> wrote:
>
> > Hello,
> >
> > Your images won't appear on the mailing list. You'll need to post them
> > elsewhere and link to them.
> >
> > On Tue, 15 Sep 2020 at 09:44, Keene Chen <[hidden email]> wrote:
> >
> > > Hi Solr users community,
> > >
> > >
> > > We have been doing some performance tests on Solr 5.5.2 and Solr 8.5.2 as
> > > part of an upgrading process, and we have noticed some reduced
> > performance
> > > for certain types of requests, particularly those that requests a large
> > > number of rows, eg. 10000. Would anyone have an explanation as to why the
> > > performance degrades, and what areas can be looked at in order to improve
> > > its performance?
> > >
> > > The performance test example below was carried out using 18000 of such
> > > queries, running at a constant throughput as specified by the label in
> > the
> > > x-axis. “Rpm” here stands for “requests per minute”.
> > >
> > > Solr 8.5’s maximum response times are consistently better. However, the
> > > 95th and 99th percentile are comparably worse than Solr 5.5’s response
> > > times.
> > > [image: image.png]
> > > [image: image.png]
> > >
> > > The maximum throughput for solr 8.5 is reached sooner than Solr 5.5 at
> > > around 4 requests per second.
> > >
> > >
> > > [image: image.png]
> > > [image: image.png]
> > >
> > > Regards,
> > > Keene
> > >
> > > --
> > >
> > >
> > > Keene Chen  | Senior Software Developer
> > >
> > >
> > >
> > > Connect with us
> > >
> > > LinkedIn <https://www.linkedin.com/company/mintel/>  Twitter
> > > <https://twitter.com/mintelnews>  Instagram
> > > <https://www.instagram.com/mintelnews/>  Facebook
> > > <https://www.facebook.com/MintelGroup/>  News
> > > <https://www.mintel.com/press-centre>  Blog <https://www.mintel.com/blog
> > >
> > >
> > > <https://www.mintel.com/>
> > >
> > >
> > >
> > > Mintel Group Ltd | 11 Pilgrim Street | London | EC4V 6RN
> > > Registered in England: Number 1475918. | VAT Number: GB 232 9342 72
> > >
> > > Contact details for our other offices can be found at
> > > http://www.mintel.com/office-locations.
> > >
> > > This email and any attachments may include content that is confidential,
> > > privileged
> > > or otherwise protected under applicable law. Unauthorised disclosure,
> > > copying, distribution
> > > or use of the contents is prohibited and may be unlawful. If you have
> > > received this email in error,
> > > including without appropriate authorisation, then please reply to the
> > > sender about the error
> > > and delete this email and any attachments.
> > >
> > >
> > >
> > > Mintel Group Ltd | 11 Pilgrim Street | London | EC4V 6RN
> > > Registered in England: Number 1475918. | VAT Number: GB 232 9342 72
> > >
> > > Contact details for our other offices can be found at
> > > http://www.mintel.com/office-locations.
> > >
> > > This email and any attachments may include content that is confidential,
> > > privileged
> > > or otherwise protected under applicable law. Unauthorised disclosure,
> > > copying, distribution
> > > or use of the contents is prohibited and may be unlawful. If you have
> > > received this email in error,
> > > including without appropriate authorisation, then please reply to the
> > > sender about the error
> > > and delete this email and any attachments.
> > >
> > >
> >
>
> --
>
> Mintel Group Ltd | 11 Pilgrim Street | London | EC4V 6RN
> Registered in
> England: Number 1475918. | VAT Number: GB 232 9342 72
>
> Contact details for
> our other offices can be found at http://www.mintel.com/office-locations
> <http://www.mintel.com/office-locations>.
>
> This email and any attachments
> may include content that is confidential, privileged
> or otherwise
> protected under applicable law. Unauthorised disclosure, copying,
> distribution
> or use of the contents is prohibited and may be unlawful. If
> you have received this email in error,
> including without appropriate
> authorisation, then please reply to the sender about the error
> and delete
> this email and any attachments.
>