Performance Issue since Solr 7.7 with wt=javabin

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Performance Issue since Solr 7.7 with wt=javabin

Andy Reek

Hi Solr team,


we are using Solr in version 7.1 as search engine in our online shop (SAP Hybris). And as a task I needed to migrate to the most recent Solr in version 7 (7.7). Doing this I faced extreme performance issues. After debugging and testing different setups I found out, that they were caused by the parameter wt=javabin. These issues begin to raise since version 7.7, in 7.6 it is still working as fast as in 7.1.


Just an example: Doing a simple query for *.* and wt=javabin in 7.6: 0.2 seconds and in 7.7: 34 seconds!


The configuration of the schema.xml and solrconfig.xml are equal in both versions. Version 8.1 has the same effect as 7.7. Using something other than wt=javabin (e.g. wt=xml) will work fast in every version - which is our current workaround.



To reproduce this issue I have attached my used configsets folder plus some test data. This all can be tested with docker and wget:


Solr 7.6:

docker run -d --name solr7.6 -p 8983:8983 --rm -v $PWD/configsets/default:/opt/solr/server/solr/configsets/myconfig:ro solr:7.6-slim solr-create -c mycore -d /opt/solr/server/solr/configsets/myconfig
docker cp $PWD/data.json solr7.6:/opt/solr/data.json
docker exec -it --user solr solr7.6 bin/post -c mycore data.json
wget "http://localhost:8983/solr/mycore/select?q=*:*&wt=javabin&rows=9999"
(0.2s)

Solr 7.7:
docker run -d --name solr7.7 -p 18983:8983 --rm -v $PWD/configsets/default:/opt/solr/server/solr/configsets/myconfig:ro solr:7.7-slim solr-create -c mycore -d /opt/solr/server/solr/configsets/myconfig
docker cp $PWD/data.json solr7.7:/opt/solr/data.json
docker exec -it --user solr solr7.7 bin/post -c mycore data.json
(34s)

For me it seems like a bug. But if not, then please let me know what I did wrong ;-)



Best Regards,

 

Andy Reek

Principal Software Developer

 

diva-e Jena

Mälzerstraße 3, 07745 Jena, Deutschland

T:       +49 (3641) 3678 (223)

F:       +49 (3641) 3678 101

[hidden email]

 

www.diva-e.com follow us: facebooktwitterLinkedIn, Xing

____________________________________________

 

diva-e AGETO GmbH

Handelsregister: HRB 210399 Amtsgericht Jena

Geschäftsführung: Sascha Sauer, Sirko Schneppe, Axel Jahn


solrConfigAndData.zip (611K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

WG: Performance Issue since Solr 7.7 with wt=javabin

Andy Reek

Hi Solr team,


we are using Solr in version 7.1 as search engine in our online shop (SAP Hybris). And as a task I needed to migrate to the most recent Solr in version 7 (7.7). Doing this I faced extreme performance issues. After debugging and testing different setups I found out, that they were caused by the parameter wt=javabin. These issues begin to raise since version 7.7, in 7.6 it is still working as fast as in 7.1.


Just an example: Doing a simple query for *.* and wt=javabin in 7.6: 0.2 seconds and in 7.7: 34 seconds!


The configuration of the schema.xml and solrconfig.xml are equal in both versions. Version 8.1 has the same effect as 7.7. Using something other than wt=javabin (e.g. wt=xml) will work fast in every version - which is our current workaround.



To reproduce this issue I have attached my used configsets folder plus some test data. This all can be tested with docker and wget:


Solr 7.6:

docker run -d --name solr7.6 -p 8983:8983 --rm -v $PWD/configsets/default:/opt/solr/server/solr/configsets/myconfig:ro solr:7.6-slim solr-create -c mycore -d /opt/solr/server/solr/configsets/myconfig
docker cp $PWD/data.json solr7.6:/opt/solr/data.json
docker exec -it --user solr solr7.6 bin/post -c mycore data.json
wget "http://localhost:8983/solr/mycore/select?q=*:*&wt=javabin&rows=9999"
(0.2s)

Solr 7.7:
docker run -d --name solr7.7 -p 18983:8983 --rm -v $PWD/configsets/default:/opt/solr/server/solr/configsets/myconfig:ro solr:7.7-slim solr-create -c mycore -d /opt/solr/server/solr/configsets/myconfig
docker cp $PWD/data.json solr7.7:/opt/solr/data.json
docker exec -it --user solr solr7.7 bin/post -c mycore data.json
(34s)

For me it seems like a bug. But if not, then please let me know what I did wrong ;-)



Best Regards,

 

Andy Reek

Principal Software Developer

 

diva-e Jena

Mälzerstraße 3, 07745 Jena, Deutschland

T:       +49 (3641) 3678 (223)

F:       +49 (3641) 3678 101

[hidden email]

 

www.diva-e.com follow us: facebooktwitterLinkedIn, Xing

____________________________________________

 

diva-e AGETO GmbH

Handelsregister: HRB 210399 Amtsgericht Jena

Geschäftsführung: Sascha Sauer, Sirko Schneppe, Axel Jahn


solrConfigAndData.zip (611K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Performance Issue since Solr 7.7 with wt=javabin

Noble Paul നോബിള്‍  नोब्ळ्
How are you consuming the output? Are you using solrj?

On Tue, Jun 18, 2019, 1:27 AM Andy Reek <[hidden email]> wrote:

> Hi Solr team,
>
>
> we are using Solr in version 7.1 as search engine in our online shop (SAP
> Hybris). And as a task I needed to migrate to the most recent Solr in
> version 7 (7.7). Doing this I faced extreme performance issues. After
> debugging and testing different setups I found out, that they were caused
> by the parameter wt=javabin. These issues begin to raise since version 7.7,
> in 7.6 it is still working as fast as in 7.1.
>
>
> Just an example: Doing a simple query for *.* and wt=javabin in 7.6: 0.2
> seconds and in 7.7: 34 seconds!
>
>
> The configuration of the schema.xml and solrconfig.xml are equal in both
> versions. Version 8.1 has the same effect as 7.7. Using something other
> than wt=javabin (e.g. wt=xml) will work fast in every version - which is
> our current workaround.
>
>
>
> To reproduce this issue I have attached my used configsets folder plus
> some test data. This all can be tested with docker and wget:
>
>
> Solr 7.6:
>
> docker run -d --name solr7.6 -p 8983:8983 --rm -v
> $PWD/configsets/default:/opt/solr/server/solr/configsets/myconfig:ro
> solr:7.6-slim solr-create -c mycore -d
> /opt/solr/server/solr/configsets/myconfig
> docker cp $PWD/data.json solr7.6:/opt/solr/data.json
> docker exec -it --user solr solr7.6 bin/post -c mycore data.json
> wget "http://localhost:8983/solr/mycore/select?q=*:*&wt=javabin&rows=9999"
> (0.2s)
>
> Solr 7.7:
> docker run -d --name solr7.7 -p 18983:8983 --rm -v
> $PWD/configsets/default:/opt/solr/server/solr/configsets/myconfig:ro
> solr:7.7-slim solr-create -c mycore -d
> /opt/solr/server/solr/configsets/myconfig
> docker cp $PWD/data.json solr7.7:/opt/solr/data.json
> docker exec -it --user solr solr7.7 bin/post -c mycore data.json
> (34s)
>
> For me it seems like a bug. But if not, then please let me know what I did
> wrong ;-)
>
>
>
> Best Regards,
>
>
>
> *Andy Reek*
>
> Principal Software Developer
>
>
>
> *diva-e* Jena
>
> Mälzerstraße 3, 07745 Jena, Deutschland
>
> T:       +49 (3641) 3678 (223)
>
> F:       +49 (3641) 3678 101
>
> *[hidden email] <[hidden email]>*
>
>
>
> *www.diva-e.com <http://www.diva-e.com>* follow us: facebook
> <https://www.facebook.com/digital.value.enterprise/?ref=hl>, twitter
> <https://twitter.com/diva_enterprise>, LinkedIn
> <https://www.linkedin.com/company/diva-e-digital-value-enterprise-gmbh>*,*
>  *Xing <https://www.xing.com/companies/diva-edigitalvalueenterprisegmbh>*
>
> ____________________________________________
>
>
>
> *diva-e* AGETO GmbH
>
> Handelsregister: HRB 210399 Amtsgericht Jena
>
> Geschäftsführung: Sascha Sauer, Sirko Schneppe, Axel Jahn
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Performance Issue since Solr 7.7 with wt=javabin

Jan Høydahl / Cominvent
In reply to this post by Andy Reek
Hi,

Did you find a solution to your performance problem?

--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com

> 17. jun. 2019 kl. 17:17 skrev Andy Reek <[hidden email]>:
>
> Hi Solr team,
>
> we are using Solr in version 7.1 as search engine in our online shop (SAP Hybris). And as a task I needed to migrate to the most recent Solr in version 7 (7.7). Doing this I faced extreme performance issues. After debugging and testing different setups I found out, that they were caused by the parameter wt=javabin. These issues begin to raise since version 7.7, in 7.6 it is still working as fast as in 7.1.
>
> Just an example: Doing a simple query for *.* and wt=javabin in 7.6: 0.2 seconds and in 7.7: 34 seconds!
>
> The configuration of the schema.xml and solrconfig.xml are equal in both versions. Version 8.1 has the same effect as 7.7. Using something other than wt=javabin (e.g. wt=xml) will work fast in every version - which is our current workaround.
>
>
> To reproduce this issue I have attached my used configsets folder plus some test data. This all can be tested with docker and wget:
>
> Solr 7.6:
> docker run -d --name solr7.6 -p 8983:8983 --rm -v $PWD/configsets/default:/opt/solr/server/solr/configsets/myconfig:ro solr:7.6-slim solr-create -c mycore -d /opt/solr/server/solr/configsets/myconfig
> docker cp $PWD/data.json solr7.6:/opt/solr/data.json
> docker exec -it --user solr solr7.6 bin/post -c mycore data.json
> wget "http://localhost:8983/solr/mycore/select?q=*:*&wt=javabin&rows=9999 <http://localhost:8983/solr/mycore/select?q=*:*&wt=javabin&rows=9999>"
> (0.2s)
>
> Solr 7.7:
> docker run -d --name solr7.7 -p 18983:8983 --rm -v $PWD/configsets/default:/opt/solr/server/solr/configsets/myconfig:ro solr:7.7-slim solr-create -c mycore -d /opt/solr/server/solr/configsets/myconfig
> docker cp $PWD/data.json solr7.7:/opt/solr/data.json
> docker exec -it --user solr solr7.7 bin/post -c mycore data.json
> (34s)
>
> For me it seems like a bug. But if not, then please let me know what I did wrong ;-)
>
>
> Best Regards,
>  
> Andy Reek
> Principal Software Developer
>  
> diva-e Jena
> Mälzerstraße 3, 07745 Jena, Deutschland
>
> T:       +49 (3641) 3678 (223)
> F:       +49 (3641) 3678 101
> [hidden email] <mailto:[hidden email]>
>  
> www.diva-e.com <http://www.diva-e.com/> follow us: facebook <https://www.facebook.com/digital.value.enterprise/?ref=hl>, twitter <https://twitter.com/diva_enterprise>, LinkedIn <https://www.linkedin.com/company/diva-e-digital-value-enterprise-gmbh>, Xing <https://www.xing.com/companies/diva-edigitalvalueenterprisegmbh>
> ____________________________________________
>  
> diva-e AGETO GmbH
> Handelsregister: HRB 210399 Amtsgericht Jena
> Geschäftsführung: Sascha Sauer, Sirko Schneppe, Axel Jahn
> <solrConfigAndData.zip>

Reply | Threaded
Open this post in threaded view
|

Re: Performance Issue since Solr 7.7 with wt=javabin

Paras Lehana
Hi Andy,

Have you run performance benchmarking for sometime and made sure that the
Solr Caching and GC doesn't impact the performance? I recommend that you
should rebuild the performance matrix after few warmups and requests. Have
you invalidated this?

On Fri, 18 Oct 2019 at 12:35, Jan Høydahl <[hidden email]> wrote:

> Hi,
>
> Did you find a solution to your performance problem?
>
> --
> Jan Høydahl, search solution architect
> Cominvent AS - www.cominvent.com
>
> > 17. jun. 2019 kl. 17:17 skrev Andy Reek <[hidden email]>:
> >
> > Hi Solr team,
> >
> > we are using Solr in version 7.1 as search engine in our online shop
> (SAP Hybris). And as a task I needed to migrate to the most recent Solr in
> version 7 (7.7). Doing this I faced extreme performance issues. After
> debugging and testing different setups I found out, that they were caused
> by the parameter wt=javabin. These issues begin to raise since version 7.7,
> in 7.6 it is still working as fast as in 7.1.
> >
> > Just an example: Doing a simple query for *.* and wt=javabin in 7.6: 0.2
> seconds and in 7.7: 34 seconds!
> >
> > The configuration of the schema.xml and solrconfig.xml are equal in both
> versions. Version 8.1 has the same effect as 7.7. Using something other
> than wt=javabin (e.g. wt=xml) will work fast in every version - which is
> our current workaround.
> >
> >
> > To reproduce this issue I have attached my used configsets folder plus
> some test data. This all can be tested with docker and wget:
> >
> > Solr 7.6:
> > docker run -d --name solr7.6 -p 8983:8983 --rm -v
> $PWD/configsets/default:/opt/solr/server/solr/configsets/myconfig:ro
> solr:7.6-slim solr-create -c mycore -d
> /opt/solr/server/solr/configsets/myconfig
> > docker cp $PWD/data.json solr7.6:/opt/solr/data.json
> > docker exec -it --user solr solr7.6 bin/post -c mycore data.json
> > wget "
> http://localhost:8983/solr/mycore/select?q=*:*&wt=javabin&rows=9999 <
> http://localhost:8983/solr/mycore/select?q=*:*&wt=javabin&rows=9999>"
> > (0.2s)
> >
> > Solr 7.7:
> > docker run -d --name solr7.7 -p 18983:8983 --rm -v
> $PWD/configsets/default:/opt/solr/server/solr/configsets/myconfig:ro
> solr:7.7-slim solr-create -c mycore -d
> /opt/solr/server/solr/configsets/myconfig
> > docker cp $PWD/data.json solr7.7:/opt/solr/data.json
> > docker exec -it --user solr solr7.7 bin/post -c mycore data.json
> > (34s)
> >
> > For me it seems like a bug. But if not, then please let me know what I
> did wrong ;-)
> >
> >
> > Best Regards,
> >
> > Andy Reek
> > Principal Software Developer
> >
> > diva-e Jena
> > Mälzerstraße 3, 07745 Jena, Deutschland
> >
> > T:       +49 (3641) 3678 (223)
> > F:       +49 (3641) 3678 101
> > [hidden email] <mailto:[hidden email]>
> >
> > www.diva-e.com <http://www.diva-e.com/> follow us: facebook <
> https://www.facebook.com/digital.value.enterprise/?ref=hl>, twitter <
> https://twitter.com/diva_enterprise>, LinkedIn <
> https://www.linkedin.com/company/diva-e-digital-value-enterprise-gmbh>,
> Xing <https://www.xing.com/companies/diva-edigitalvalueenterprisegmbh>
> > ____________________________________________
> >
> > diva-e AGETO GmbH
> > Handelsregister: HRB 210399 Amtsgericht Jena
> > Geschäftsführung: Sascha Sauer, Sirko Schneppe, Axel Jahn
> > <solrConfigAndData.zip>
>
>

--
--
Regards,

*Paras Lehana* [65871]
Software Programmer, Auto-Suggest,
IndiaMART Intermesh Ltd.

8th Floor, Tower A, Advant-Navis Business Park, Sector 142,
Noida, UP, IN - 201303

Mob.: +91-9560911996
Work: 01203916600 | Extn:  *8173*

--
IMPORTANT: 
NEVER share your IndiaMART OTP/ Password with anyone.