Solr LTR model Performance Issues

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Solr LTR model Performance Issues

kamal11
Hi,

I am trying to use LTR with solr 6.6.2.There are different types of model
like Linear Model, Multiple Additive Trees Model and Neural Network Model.

I have tried using Linear & Multiadditive model and compared the
performance of results. There is a major difference in response time
between the 2 models. I am observing that Multiadditive model is taking way
higher time than linear model.

Is there a way we can improve the performance here.

Note: The size of Multiadditive model is 136 MB.

Regards
Kamal Kishore

<https://www.avast.com/en-in/recommend?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=default3&tag=d4ef6ef9-b8d1-40b8-96ac-2354fd69483b>
I’m
protected online with Avast Free Antivirus. Get it here — it’s free forever.
<https://www.avast.com/en-in/recommend?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=default3&tag=d4ef6ef9-b8d1-40b8-96ac-2354fd69483b>
<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
Reply | Threaded
Open this post in threaded view
|

Re: Solr LTR model Performance Issues

Jörn Franke
Can you share the time needed of the two models? How many documents? What is your loading pipeline? Have you observed cpu/memory?

> Am 22.03.2019 um 12:01 schrieb Kamal Kishore Aggarwal <[hidden email]>:
>
> Hi,
>
> I am trying to use LTR with solr 6.6.2.There are different types of model
> like Linear Model, Multiple Additive Trees Model and Neural Network Model.
>
> I have tried using Linear & Multiadditive model and compared the
> performance of results. There is a major difference in response time
> between the 2 models. I am observing that Multiadditive model is taking way
> higher time than linear model.
>
> Is there a way we can improve the performance here.
>
> Note: The size of Multiadditive model is 136 MB.
>
> Regards
> Kamal Kishore
>
> <https://www.avast.com/en-in/recommend?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=default3&tag=d4ef6ef9-b8d1-40b8-96ac-2354fd69483b>
> I’m
> protected online with Avast Free Antivirus. Get it here — it’s free forever.
> <https://www.avast.com/en-in/recommend?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=default3&tag=d4ef6ef9-b8d1-40b8-96ac-2354fd69483b>
> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
Reply | Threaded
Open this post in threaded view
|

Re: Solr LTR model Performance Issues

kamal11
HI Jörn Franke,

Thanks for the quick reply.

I have performed the jmeter load testing on one of the server for Linear vs
Multipleadditive tree model. We are using lucidworks fusion.
There is some business logic in the query pipeline followed by main solr
ltr query. This is the total time taken by query pipeline.
Below are the response time:

# of Threads Ramup Period Loop Count Type Total Requests Average Response
Time (ms)
Iteration 1 Iteration 2 Iteration 3
10 1 10 Linear Model  100 2038 1998 1975
25 1 10 Linear Model  250 4329 3961 3726

10 1 10 MultiAdditive Model 100 12721 12631 12567
25 1 10 MultiAdditive Model 250 27924 31420 30758
# of docs: 500K and Indexing size is 10 GB.

As of now, I did not checked the CPU or memory usage, but did not observed
any errors during jmeter load test.

Let me know if any other information is required.

Regards
Kamal

<https://www.avast.com/en-in/recommend?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=default3&tag=d4ef6ef9-b8d1-40b8-96ac-2354fd69483b>
I’m
protected online with Avast Free Antivirus. Get it here — it’s free forever.
<https://www.avast.com/en-in/recommend?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=default3&tag=d4ef6ef9-b8d1-40b8-96ac-2354fd69483b>
<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

On Fri, Mar 22, 2019 at 5:13 PM Jörn Franke <[hidden email]> wrote:

> Can you share the time needed of the two models? How many documents? What
> is your loading pipeline? Have you observed cpu/memory?
>
> > Am 22.03.2019 um 12:01 schrieb Kamal Kishore Aggarwal <
> [hidden email]>:
> >
> > Hi,
> >
> > I am trying to use LTR with solr 6.6.2.There are different types of model
> > like Linear Model, Multiple Additive Trees Model and Neural Network
> Model.
> >
> > I have tried using Linear & Multiadditive model and compared the
> > performance of results. There is a major difference in response time
> > between the 2 models. I am observing that Multiadditive model is taking
> way
> > higher time than linear model.
> >
> > Is there a way we can improve the performance here.
> >
> > Note: The size of Multiadditive model is 136 MB.
> >
> > Regards
> > Kamal Kishore
> >
> > <
> https://www.avast.com/en-in/recommend?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=default3&tag=d4ef6ef9-b8d1-40b8-96ac-2354fd69483b
> >
> > I’m
> > protected online with Avast Free Antivirus. Get it here — it’s free
> forever.
> > <
> https://www.avast.com/en-in/recommend?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=default3&tag=d4ef6ef9-b8d1-40b8-96ac-2354fd69483b
> >
> > <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>
Reply | Threaded
Open this post in threaded view
|

Re: Solr LTR model Performance Issues

kamal11
Hi,

Any update on this?
Is this model running in multi threaded mode or is there is any scope to do
this. Please let me know.

Regards
Kamal

On Sat, Mar 23, 2019 at 10:35 AM Kamal Kishore Aggarwal <
[hidden email]> wrote:

> HI Jörn Franke,
>
> Thanks for the quick reply.
>
> I have performed the jmeter load testing on one of the server for Linear
> vs Multipleadditive tree model. We are using lucidworks fusion.
> There is some business logic in the query pipeline followed by main solr
> ltr query. This is the total time taken by query pipeline.
> Below are the response time:
>
> # of Threads Ramup Period Loop Count Type Total Requests Average Response
> Time (ms)
> Iteration 1 Iteration 2 Iteration 3
> 10 1 10 Linear Model  100 2038 1998 1975
> 25 1 10 Linear Model  250 4329 3961 3726
>
> 10 1 10 MultiAdditive Model 100 12721 12631 12567
> 25 1 10 MultiAdditive Model 250 27924 31420 30758
> # of docs: 500K and Indexing size is 10 GB.
>
> As of now, I did not checked the CPU or memory usage, but did not observed
> any errors during jmeter load test.
>
> Let me know if any other information is required.
>
> Regards
> Kamal
>
>
> <https://www.avast.com/en-in/recommend?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=default3&tag=d4ef6ef9-b8d1-40b8-96ac-2354fd69483b> I’m
> protected online with Avast Free Antivirus. Get it here — it’s free
> forever.
> <https://www.avast.com/en-in/recommend?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=default3&tag=d4ef6ef9-b8d1-40b8-96ac-2354fd69483b>
> <#m_-1438210790161476832_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>
> On Fri, Mar 22, 2019 at 5:13 PM Jörn Franke <[hidden email]> wrote:
>
>> Can you share the time needed of the two models? How many documents? What
>> is your loading pipeline? Have you observed cpu/memory?
>>
>> > Am 22.03.2019 um 12:01 schrieb Kamal Kishore Aggarwal <
>> [hidden email]>:
>> >
>> > Hi,
>> >
>> > I am trying to use LTR with solr 6.6.2.There are different types of
>> model
>> > like Linear Model, Multiple Additive Trees Model and Neural Network
>> Model.
>> >
>> > I have tried using Linear & Multiadditive model and compared the
>> > performance of results. There is a major difference in response time
>> > between the 2 models. I am observing that Multiadditive model is taking
>> way
>> > higher time than linear model.
>> >
>> > Is there a way we can improve the performance here.
>> >
>> > Note: The size of Multiadditive model is 136 MB.
>> >
>> > Regards
>> > Kamal Kishore
>> >
>> > <
>> https://www.avast.com/en-in/recommend?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=default3&tag=d4ef6ef9-b8d1-40b8-96ac-2354fd69483b
>> >
>> > I’m
>> > protected online with Avast Free Antivirus. Get it here — it’s free
>> forever.
>> > <
>> https://www.avast.com/en-in/recommend?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=default3&tag=d4ef6ef9-b8d1-40b8-96ac-2354fd69483b
>> >
>> > <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>>
>
Reply | Threaded
Open this post in threaded view
|

Re: Solr LTR model Performance Issues

Jörn Franke
It is a little bit difficult to say, because it could be also the business logic in the query execution. What is your performance baseline, ie if you just execute one query for each of the models?
How fast should it be? Do you have really 10 or more concurrent users, or users that fire up queries at exactly the same time?

Can you please monitor CPU and memory?

> Am 05.04.2019 um 21:42 schrieb Kamal Kishore Aggarwal <[hidden email]>:
>
> Hi,
>
> Any update on this?
> Is this model running in multi threaded mode or is there is any scope to do
> this. Please let me know.
>
> Regards
> Kamal
>
> On Sat, Mar 23, 2019 at 10:35 AM Kamal Kishore Aggarwal <
> [hidden email]> wrote:
>
>> HI Jörn Franke,
>>
>> Thanks for the quick reply.
>>
>> I have performed the jmeter load testing on one of the server for Linear
>> vs Multipleadditive tree model. We are using lucidworks fusion.
>> There is some business logic in the query pipeline followed by main solr
>> ltr query. This is the total time taken by query pipeline.
>> Below are the response time:
>>
>> # of Threads Ramup Period Loop Count Type Total Requests Average Response
>> Time (ms)
>> Iteration 1 Iteration 2 Iteration 3
>> 10 1 10 Linear Model  100 2038 1998 1975
>> 25 1 10 Linear Model  250 4329 3961 3726
>>
>> 10 1 10 MultiAdditive Model 100 12721 12631 12567
>> 25 1 10 MultiAdditive Model 250 27924 31420 30758
>> # of docs: 500K and Indexing size is 10 GB.
>>
>> As of now, I did not checked the CPU or memory usage, but did not observed
>> any errors during jmeter load test.
>>
>> Let me know if any other information is required.
>>
>> Regards
>> Kamal
>>
>>
>> <https://www.avast.com/en-in/recommend?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=default3&tag=d4ef6ef9-b8d1-40b8-96ac-2354fd69483b> I’m
>> protected online with Avast Free Antivirus. Get it here — it’s free
>> forever.
>> <https://www.avast.com/en-in/recommend?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=default3&tag=d4ef6ef9-b8d1-40b8-96ac-2354fd69483b>
>> <#m_-1438210790161476832_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>>
>>> On Fri, Mar 22, 2019 at 5:13 PM Jörn Franke <[hidden email]> wrote:
>>>
>>> Can you share the time needed of the two models? How many documents? What
>>> is your loading pipeline? Have you observed cpu/memory?
>>>
>>>> Am 22.03.2019 um 12:01 schrieb Kamal Kishore Aggarwal <
>>> [hidden email]>:
>>>>
>>>> Hi,
>>>>
>>>> I am trying to use LTR with solr 6.6.2.There are different types of
>>> model
>>>> like Linear Model, Multiple Additive Trees Model and Neural Network
>>> Model.
>>>>
>>>> I have tried using Linear & Multiadditive model and compared the
>>>> performance of results. There is a major difference in response time
>>>> between the 2 models. I am observing that Multiadditive model is taking
>>> way
>>>> higher time than linear model.
>>>>
>>>> Is there a way we can improve the performance here.
>>>>
>>>> Note: The size of Multiadditive model is 136 MB.
>>>>
>>>> Regards
>>>> Kamal Kishore
>>>>
>>>> <
>>> https://www.avast.com/en-in/recommend?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=default3&tag=d4ef6ef9-b8d1-40b8-96ac-2354fd69483b
>>>>
>>>> I’m
>>>> protected online with Avast Free Antivirus. Get it here — it’s free
>>> forever.
>>>> <
>>> https://www.avast.com/en-in/recommend?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=default3&tag=d4ef6ef9-b8d1-40b8-96ac-2354fd69483b
>>>>
>>>> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>>>
>>
Reply | Threaded
Open this post in threaded view
|

Re: Solr LTR model Performance Issues

kamal11
Hi,

I made change in the model by making the LTRScoringModel as immutable and
cache hashCode calculation. The response time improved a lot after the
change.

http://lucene.472066.n3.nabble.com/jira-Updated-SOLR-12688-LTR-Multiple-performance-fixes-pure-DocValues-support-for-FieldValueFeature-td4404254.html


On Sat, Apr 6, 2019 at 12:22 PM Jörn Franke <[hidden email]> wrote:

> It is a little bit difficult to say, because it could be also the business
> logic in the query execution. What is your performance baseline, ie if you
> just execute one query for each of the models?
> How fast should it be? Do you have really 10 or more concurrent users, or
> users that fire up queries at exactly the same time?
>
> Can you please monitor CPU and memory?
>
> > Am 05.04.2019 um 21:42 schrieb Kamal Kishore Aggarwal <
> [hidden email]>:
> >
> > Hi,
> >
> > Any update on this?
> > Is this model running in multi threaded mode or is there is any scope to
> do
> > this. Please let me know.
> >
> > Regards
> > Kamal
> >
> > On Sat, Mar 23, 2019 at 10:35 AM Kamal Kishore Aggarwal <
> > [hidden email]> wrote:
> >
> >> HI Jörn Franke,
> >>
> >> Thanks for the quick reply.
> >>
> >> I have performed the jmeter load testing on one of the server for Linear
> >> vs Multipleadditive tree model. We are using lucidworks fusion.
> >> There is some business logic in the query pipeline followed by main solr
> >> ltr query. This is the total time taken by query pipeline.
> >> Below are the response time:
> >>
> >> # of Threads Ramup Period Loop Count Type Total Requests Average
> Response
> >> Time (ms)
> >> Iteration 1 Iteration 2 Iteration 3
> >> 10 1 10 Linear Model  100 2038 1998 1975
> >> 25 1 10 Linear Model  250 4329 3961 3726
> >>
> >> 10 1 10 MultiAdditive Model 100 12721 12631 12567
> >> 25 1 10 MultiAdditive Model 250 27924 31420 30758
> >> # of docs: 500K and Indexing size is 10 GB.
> >>
> >> As of now, I did not checked the CPU or memory usage, but did not
> observed
> >> any errors during jmeter load test.
> >>
> >> Let me know if any other information is required.
> >>
> >> Regards
> >> Kamal
> >>
> >>
> >> <
> https://www.avast.com/en-in/recommend?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=default3&tag=d4ef6ef9-b8d1-40b8-96ac-2354fd69483b>
> I’m
> >> protected online with Avast Free Antivirus. Get it here — it’s free
> >> forever.
> >> <
> https://www.avast.com/en-in/recommend?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=default3&tag=d4ef6ef9-b8d1-40b8-96ac-2354fd69483b
> >
> >> <#m_-1438210790161476832_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
> >>
> >>> On Fri, Mar 22, 2019 at 5:13 PM Jörn Franke <[hidden email]>
> wrote:
> >>>
> >>> Can you share the time needed of the two models? How many documents?
> What
> >>> is your loading pipeline? Have you observed cpu/memory?
> >>>
> >>>> Am 22.03.2019 um 12:01 schrieb Kamal Kishore Aggarwal <
> >>> [hidden email]>:
> >>>>
> >>>> Hi,
> >>>>
> >>>> I am trying to use LTR with solr 6.6.2.There are different types of
> >>> model
> >>>> like Linear Model, Multiple Additive Trees Model and Neural Network
> >>> Model.
> >>>>
> >>>> I have tried using Linear & Multiadditive model and compared the
> >>>> performance of results. There is a major difference in response time
> >>>> between the 2 models. I am observing that Multiadditive model is
> taking
> >>> way
> >>>> higher time than linear model.
> >>>>
> >>>> Is there a way we can improve the performance here.
> >>>>
> >>>> Note: The size of Multiadditive model is 136 MB.
> >>>>
> >>>> Regards
> >>>> Kamal Kishore
> >>>>
> >>>> <
> >>>
> https://www.avast.com/en-in/recommend?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=default3&tag=d4ef6ef9-b8d1-40b8-96ac-2354fd69483b
> >>>>
> >>>> I’m
> >>>> protected online with Avast Free Antivirus. Get it here — it’s free
> >>> forever.
> >>>> <
> >>>
> https://www.avast.com/en-in/recommend?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=default3&tag=d4ef6ef9-b8d1-40b8-96ac-2354fd69483b
> >>>>
> >>>> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
> >>>
> >>
>