Solr query takes a too much time in Solr 6.1.0

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Solr query takes a too much time in Solr 6.1.0

vishal patel-2
We have 2 shards and 2 replicas in Live environment. we have multiple collections.
Some times some query takes much time(QTime=52552).  There are so many documents indexing and searching within milliseconds.
When we executed the same query again using admin panel, it does not take a much time and it completes within 20 milliseconds.

My Solr Logs :
2019-05-10 09:48:56.744 INFO  (qtp1239731077-128223) [c:actionscomments s:shard1 r:core_node1 x:actionscomments] o.a.s.c.S.Request [actionscomments]  webapp=/solr path=/select params={q=%2Bproject_id:(2102117)%2Brecipient_id:(4642365)+%2Bentity_type:(1)+-action_id:(20+32)+%2Baction_status:(0)+%2Bis_active:(true)+%2B(is_formtype_active:true)+%2B(appType:1)&shards=s1.example.com:8983/solr/actionscomments|s1r1.example.com:8983/solr/actionscomments,s2.example.com:8983/solr/actionscomments|s2r1.example.com:8983/solr/actionscomments&indent=off&shards.tolerant=true&fl=id&start=0&sort=id+desc,id+desc&fq=&rows=1} hits=198 status=0 QTime=52552
2019-05-10 09:48:56.744 INFO  (qtp1239731077-127998) [c:actionscomments s:shard1 r:core_node1 x:actionscomments] o.a.s.c.S.Request [actionscomments]  webapp=/solr path=/select params={q=%2Bproject_id:(2102117)%2Brecipient_id:(4642365)+%2Bentity_type:(1)+-action_id:(20+32)+%2Baction_status:(0)+%2Bis_active:(true)+%2Bdue_date:[2019-05-09T19:30:00Z+TO+2019-05-09T19:30:00Z%2B1DAY]+%2B(is_formtype_active:true)+%2B(appType:1)&shards=s1.example.com:8983/solr/actionscomments|s1r1.example.com:8983/solr/actionscomments,s2.example.com:8983/solr/actionscomments|s2r1.example.com:8983/solr/actionscomments&indent=off&shards.tolerant=true&fl=id&start=0&sort=id+desc,id+desc&fq=&rows=1} hits=0 status=0 QTime=51970
2019-05-10 09:48:56.746 INFO  (qtp1239731077-128224) [c:actionscomments s:shard1 r:core_node1 x:actionscomments] o.a.s.c.S.Request [actionscomments]  webapp=/solr path=/select params={q=%2Bproject_id:(2121600+2115171+2104206)%2Brecipient_id:(2834330)+%2Bentity_type:(2)+-action_id:(20+32)+%2Baction_status:(0)+%2Bis_active:(true)+%2Bdue_date:[2019-05-10T00:00:00Z+TO+2019-05-10T00:00:00Z%2B1DAY]&shards=s1.example.com:8983/solr/actionscomments|s1r1.example.com:8983/solr/actionscomments,s2.example.com:8983/solr/actionscomments|s2r1.example.com:8983/solr/actionscomments&indent=off&shards.tolerant=true&fl=id&start=0&sort=id+desc,id+desc&fq=&rows=1} hits=98 status=0 QTime=51402


My schema fields below :

<field name="id" type="string" indexed="true" stored="true" required="true" multiValued="false"/>
<field name="project_id" type="tint" indexed="true" stored="true"/>
<field name="recipient_id" type="tint" indexed="true" stored="true"/>
<field name="entity_type" type="tint" indexed="true" stored="true"/>
<field name="action_id" type="tint" indexed="true" stored="true"/>
<field name="action_status" type="tint" indexed="true" stored="true"/>
<field name="is_active" type="boolean" indexed="true" stored="true" />
<field name="is_formtype_active" type="boolean" indexed="true" stored="true" />
<field name="appType" type="tint" indexed="true" stored="true" />
<field name="due_date" type="date" indexed="true" stored="true"/>

What could be a problem here? why the query takes too much time at that time?

Sent from Outlook<http://aka.ms/weboutlook>
Reply | Threaded
Open this post in threaded view
|

Re: Solr query takes a too much time in Solr 6.1.0

David Hastings
first inclination is your index is cold.

On Fri, May 10, 2019 at 9:32 AM vishal patel <[hidden email]>
wrote:

> We have 2 shards and 2 replicas in Live environment. we have multiple
> collections.
> Some times some query takes much time(QTime=52552).  There are so many
> documents indexing and searching within milliseconds.
> When we executed the same query again using admin panel, it does not take
> a much time and it completes within 20 milliseconds.
>
> My Solr Logs :
> 2019-05-10 09:48:56.744 INFO  (qtp1239731077-128223) [c:actionscomments
> s:shard1 r:core_node1 x:actionscomments] o.a.s.c.S.Request
> [actionscomments]  webapp=/solr path=/select
> params={q=%2Bproject_id:(2102117)%2Brecipient_id:(4642365)+%2Bentity_type:(1)+-action_id:(20+32)+%2Baction_status:(0)+%2Bis_active:(true)+%2B(is_formtype_active:true)+%2B(appType:1)&shards=
> s1.example.com:8983/solr/actionscomments|s1r1.example.com:8983/solr/actionscomments,s2.example.com:8983/solr/actionscomments|s2r1.example.com:8983/solr/actionscomments&indent=off&shards.tolerant=true&fl=id&start=0&sort=id+desc,id+desc&fq=&rows=1
> <http://s1.example.com:8983/solr/actionscomments%7Cs1r1.example.com:8983/solr/actionscomments,s2.example.com:8983/solr/actionscomments%7Cs2r1.example.com:8983/solr/actionscomments&indent=off&shards.tolerant=true&fl=id&start=0&sort=id+desc,id+desc&fq=&rows=1>}
> hits=198 status=0 QTime=52552
> 2019-05-10 09:48:56.744 INFO  (qtp1239731077-127998) [c:actionscomments
> s:shard1 r:core_node1 x:actionscomments] o.a.s.c.S.Request
> [actionscomments]  webapp=/solr path=/select
> params={q=%2Bproject_id:(2102117)%2Brecipient_id:(4642365)+%2Bentity_type:(1)+-action_id:(20+32)+%2Baction_status:(0)+%2Bis_active:(true)+%2Bdue_date:[2019-05-09T19:30:00Z+TO+2019-05-09T19:30:00Z%2B1DAY]+%2B(is_formtype_active:true)+%2B(appType:1)&shards=
> s1.example.com:8983/solr/actionscomments|s1r1.example.com:8983/solr/actionscomments,s2.example.com:8983/solr/actionscomments|s2r1.example.com:8983/solr/actionscomments&indent=off&shards.tolerant=true&fl=id&start=0&sort=id+desc,id+desc&fq=&rows=1
> <http://s1.example.com:8983/solr/actionscomments%7Cs1r1.example.com:8983/solr/actionscomments,s2.example.com:8983/solr/actionscomments%7Cs2r1.example.com:8983/solr/actionscomments&indent=off&shards.tolerant=true&fl=id&start=0&sort=id+desc,id+desc&fq=&rows=1>}
> hits=0 status=0 QTime=51970
> 2019-05-10 09:48:56.746 INFO  (qtp1239731077-128224) [c:actionscomments
> s:shard1 r:core_node1 x:actionscomments] o.a.s.c.S.Request
> [actionscomments]  webapp=/solr path=/select
> params={q=%2Bproject_id:(2121600+2115171+2104206)%2Brecipient_id:(2834330)+%2Bentity_type:(2)+-action_id:(20+32)+%2Baction_status:(0)+%2Bis_active:(true)+%2Bdue_date:[2019-05-10T00:00:00Z+TO+2019-05-10T00:00:00Z%2B1DAY]&shards=
> s1.example.com:8983/solr/actionscomments|s1r1.example.com:8983/solr/actionscomments,s2.example.com:8983/solr/actionscomments|s2r1.example.com:8983/solr/actionscomments&indent=off&shards.tolerant=true&fl=id&start=0&sort=id+desc,id+desc&fq=&rows=1
> <http://s1.example.com:8983/solr/actionscomments%7Cs1r1.example.com:8983/solr/actionscomments,s2.example.com:8983/solr/actionscomments%7Cs2r1.example.com:8983/solr/actionscomments&indent=off&shards.tolerant=true&fl=id&start=0&sort=id+desc,id+desc&fq=&rows=1>}
> hits=98 status=0 QTime=51402
>
>
> My schema fields below :
>
> <field name="id" type="string" indexed="true" stored="true"
> required="true" multiValued="false"/>
> <field name="project_id" type="tint" indexed="true" stored="true"/>
> <field name="recipient_id" type="tint" indexed="true" stored="true"/>
> <field name="entity_type" type="tint" indexed="true" stored="true"/>
> <field name="action_id" type="tint" indexed="true" stored="true"/>
> <field name="action_status" type="tint" indexed="true" stored="true"/>
> <field name="is_active" type="boolean" indexed="true" stored="true" />
> <field name="is_formtype_active" type="boolean" indexed="true"
> stored="true" />
> <field name="appType" type="tint" indexed="true" stored="true" />
> <field name="due_date" type="date" indexed="true" stored="true"/>
>
> What could be a problem here? why the query takes too much time at that
> time?
>
> Sent from Outlook<http://aka.ms/weboutlook>
>
Reply | Threaded
Open this post in threaded view
|

Re: Solr query takes a too much time in Solr 6.1.0

Shawn Heisey-2
In reply to this post by vishal patel-2
On 5/10/2019 7:32 AM, vishal patel wrote:
> We have 2 shards and 2 replicas in Live environment. we have multiple collections.
> Some times some query takes much time(QTime=52552).  There are so many documents indexing and searching within milliseconds.

There could be any number of causes of slow performance.

A common reason is not having enough spare memory in the machine to
allow the operating system to cache the index data.  This is memory NOT
allocated by programs (including Solr).

Another common reason is that the heap size is too small, which causes
Java to frequently perform full garbage collections, which will REALLY
kill performance.

Since there's very little information here, it's difficult for us to
diagnose the cause.  Here's a wiki page about performance problems:

https://wiki.apache.org/solr/SolrPerformanceProblems

(Disclaimer:  I am the principal author of that page)

> When we executed the same query again using admin panel, it does not take a much time and it completes within 20 milliseconds.

Executing an identical query again will likely satisfy the query from
Solr's caches.  Solr won't need to talk to the actual index, and it will
be REALLY fast.  Even a massively complex query, if it is cached, will
be fast.

Running the information from your logs through a URL decoder, this is
what I found:

q=+project_id:(2102117)+recipient_id:(4642365) +entity_type:(1)
-action_id:(20 32) +action_status:(0) +is_active:(true)
+(is_formtype_active:true) +(appType:1)

If all of those fields are indexed, then I would not expect a properly
sized server to be slow.  If any of those fields are indexed=false and
have docValues, then it could be a schema configuration issue.
Searching docValues does work, but it's really slow.

Your query does have an empty fq ... "fq=" ... I do not know whether
that's problematic.  Try it without that to verify.  I would not expect
it to cause problems, but I can't be sure.

Thanks,
Shawn
Reply | Threaded
Open this post in threaded view
|

Re: Solr query takes a too much time in Solr 6.1.0

Bernd Fehling
In reply to this post by vishal patel-2
Your "sort" parameter has "sort=id+desc,id+desc".
1. It doesn't make sense to have a sort on "id" in descending order twice.
2. Be aware that the id field has the highest cadinality.
3. To speedup sorting have a separate field with docValues=true for sorting.
    E.g.
<field name="id" type="string" indexed="true" stored="true" required="true" multiValued="false" />
<field name="id_sort" type="string" indexed="false" stored="false" docValues="true" useDocValuesAsStored="false" />
<copyField source="id" dest="id_sort" />

Regards
Bernd


Am 10.05.19 um 15:32 schrieb vishal patel:

> We have 2 shards and 2 replicas in Live environment. we have multiple collections.
> Some times some query takes much time(QTime=52552).  There are so many documents indexing and searching within milliseconds.
> When we executed the same query again using admin panel, it does not take a much time and it completes within 20 milliseconds.
>
> My Solr Logs :
> 2019-05-10 09:48:56.744 INFO  (qtp1239731077-128223) [c:actionscomments s:shard1 r:core_node1 x:actionscomments] o.a.s.c.S.Request [actionscomments]  webapp=/solr path=/select params={q=%2Bproject_id:(2102117)%2Brecipient_id:(4642365)+%2Bentity_type:(1)+-action_id:(20+32)+%2Baction_status:(0)+%2Bis_active:(true)+%2B(is_formtype_active:true)+%2B(appType:1)&shards=s1.example.com:8983/solr/actionscomments|s1r1.example.com:8983/solr/actionscomments,s2.example.com:8983/solr/actionscomments|s2r1.example.com:8983/solr/actionscomments&indent=off&shards.tolerant=true&fl=id&start=0&sort=id+desc,id+desc&fq=&rows=1} hits=198 status=0 QTime=52552
> 2019-05-10 09:48:56.744 INFO  (qtp1239731077-127998) [c:actionscomments s:shard1 r:core_node1 x:actionscomments] o.a.s.c.S.Request [actionscomments]  webapp=/solr path=/select params={q=%2Bproject_id:(2102117)%2Brecipient_id:(4642365)+%2Bentity_type:(1)+-action_id:(20+32)+%2Baction_status:(0)+%2Bis_active:(true)+%2Bdue_date:[2019-05-09T19:30:00Z+TO+2019-05-09T19:30:00Z%2B1DAY]+%2B(is_formtype_active:true)+%2B(appType:1)&shards=s1.example.com:8983/solr/actionscomments|s1r1.example.com:8983/solr/actionscomments,s2.example.com:8983/solr/actionscomments|s2r1.example.com:8983/solr/actionscomments&indent=off&shards.tolerant=true&fl=id&start=0&sort=id+desc,id+desc&fq=&rows=1} hits=0 status=0 QTime=51970
> 2019-05-10 09:48:56.746 INFO  (qtp1239731077-128224) [c:actionscomments s:shard1 r:core_node1 x:actionscomments] o.a.s.c.S.Request [actionscomments]  webapp=/solr path=/select params={q=%2Bproject_id:(2121600+2115171+2104206)%2Brecipient_id:(2834330)+%2Bentity_type:(2)+-action_id:(20+32)+%2Baction_status:(0)+%2Bis_active:(true)+%2Bdue_date:[2019-05-10T00:00:00Z+TO+2019-05-10T00:00:00Z%2B1DAY]&shards=s1.example.com:8983/solr/actionscomments|s1r1.example.com:8983/solr/actionscomments,s2.example.com:8983/solr/actionscomments|s2r1.example.com:8983/solr/actionscomments&indent=off&shards.tolerant=true&fl=id&start=0&sort=id+desc,id+desc&fq=&rows=1} hits=98 status=0 QTime=51402
>
>
> My schema fields below :
>
> <field name="id" type="string" indexed="true" stored="true" required="true" multiValued="false"/>
> <field name="project_id" type="tint" indexed="true" stored="true"/>
> <field name="recipient_id" type="tint" indexed="true" stored="true"/>
> <field name="entity_type" type="tint" indexed="true" stored="true"/>
> <field name="action_id" type="tint" indexed="true" stored="true"/>
> <field name="action_status" type="tint" indexed="true" stored="true"/>
> <field name="is_active" type="boolean" indexed="true" stored="true" />
> <field name="is_formtype_active" type="boolean" indexed="true" stored="true" />
> <field name="appType" type="tint" indexed="true" stored="true" />
> <field name="due_date" type="date" indexed="true" stored="true"/>
>
> What could be a problem here? why the query takes too much time at that time?
>
> Sent from Outlook<http://aka.ms/weboutlook>
>
Reply | Threaded
Open this post in threaded view
|

Re: Solr query takes a too much time in Solr 6.1.0

vishal patel-2
In reply to this post by Shawn Heisey-2
Thanks for the reply.

> Executing an identical query again will likely satisfy the query from Solr's caches.  Solr won't need to talk to the actual index, and it will be REALLY fast.  Even a massively complex query, if it is cached, will be fast.

All caches are disabled in our schema file because of our indexing and searching ratio is high in our live environment.


Sent from Outlook<http://aka.ms/weboutlook>
________________________________
From: Shawn Heisey <[hidden email]>
Sent: Friday, May 10, 2019 9:32 PM
To: [hidden email]
Subject: Re: Solr query takes a too much time in Solr 6.1.0

On 5/10/2019 7:32 AM, vishal patel wrote:
> We have 2 shards and 2 replicas in Live environment. we have multiple collections.
> Some times some query takes much time(QTime=52552).  There are so many documents indexing and searching within milliseconds.

There could be any number of causes of slow performance.

A common reason is not having enough spare memory in the machine to
allow the operating system to cache the index data.  This is memory NOT
allocated by programs (including Solr).

Another common reason is that the heap size is too small, which causes
Java to frequently perform full garbage collections, which will REALLY
kill performance.

Since there's very little information here, it's difficult for us to
diagnose the cause.  Here's a wiki page about performance problems:

https://wiki.apache.org/solr/SolrPerformanceProblems

(Disclaimer:  I am the principal author of that page)

> When we executed the same query again using admin panel, it does not take a much time and it completes within 20 milliseconds.

Executing an identical query again will likely satisfy the query from
Solr's caches.  Solr won't need to talk to the actual index, and it will
be REALLY fast.  Even a massively complex query, if it is cached, will
be fast.

Running the information from your logs through a URL decoder, this is
what I found:

q=+project_id:(2102117)+recipient_id:(4642365) +entity_type:(1)
-action_id:(20 32) +action_status:(0) +is_active:(true)
+(is_formtype_active:true) +(appType:1)

If all of those fields are indexed, then I would not expect a properly
sized server to be slow.  If any of those fields are indexed=false and
have docValues, then it could be a schema configuration issue.
Searching docValues does work, but it's really slow.

Your query does have an empty fq ... "fq=" ... I do not know whether
that's problematic.  Try it without that to verify.  I would not expect
it to cause problems, but I can't be sure.

Thanks,
Shawn
Reply | Threaded
Open this post in threaded view
|

Re: Solr query takes a too much time in Solr 6.1.0

vishal patel-2
In reply to this post by Bernd Fehling
In our live environment, there are many searching and indexing within a millisecond. we used facet and sorting in Query.

> 3. To speedup sorting have a separate field with docValues=true for sorting.

Is it necessary or useful to make a separate field if I used this field in sorting or facet?
If I do not do a separate field then any performance issue when the same field will search in a query?

Sent from Outlook<http://aka.ms/weboutlook>
________________________________
From: Bernd Fehling <[hidden email]>
Sent: Monday, May 13, 2019 11:52 AM
To: [hidden email]
Subject: Re: Solr query takes a too much time in Solr 6.1.0

Your "sort" parameter has "sort=id+desc,id+desc".
1. It doesn't make sense to have a sort on "id" in descending order twice.
2. Be aware that the id field has the highest cadinality.
3. To speedup sorting have a separate field with docValues=true for sorting.
    E.g.
<field name="id" type="string" indexed="true" stored="true" required="true" multiValued="false" />
<field name="id_sort" type="string" indexed="false" stored="false" docValues="true" useDocValuesAsStored="false" />
<copyField source="id" dest="id_sort" />

Regards
Bernd


Am 10.05.19 um 15:32 schrieb vishal patel:

> We have 2 shards and 2 replicas in Live environment. we have multiple collections.
> Some times some query takes much time(QTime=52552).  There are so many documents indexing and searching within milliseconds.
> When we executed the same query again using admin panel, it does not take a much time and it completes within 20 milliseconds.
>
> My Solr Logs :
> 2019-05-10 09:48:56.744 INFO  (qtp1239731077-128223) [c:actionscomments s:shard1 r:core_node1 x:actionscomments] o.a.s.c.S.Request [actionscomments]  webapp=/solr path=/select params={q=%2Bproject_id:(2102117)%2Brecipient_id:(4642365)+%2Bentity_type:(1)+-action_id:(20+32)+%2Baction_status:(0)+%2Bis_active:(true)+%2B(is_formtype_active:true)+%2B(appType:1)&shards=s1.example.com:8983/solr/actionscomments|s1r1.example.com:8983/solr/actionscomments,s2.example.com:8983/solr/actionscomments|s2r1.example.com:8983/solr/actionscomments&indent=off&shards.tolerant=true&fl=id&start=0&sort=id+desc,id+desc&fq=&rows=1} hits=198 status=0 QTime=52552
> 2019-05-10 09:48:56.744 INFO  (qtp1239731077-127998) [c:actionscomments s:shard1 r:core_node1 x:actionscomments] o.a.s.c.S.Request [actionscomments]  webapp=/solr path=/select params={q=%2Bproject_id:(2102117)%2Brecipient_id:(4642365)+%2Bentity_type:(1)+-action_id:(20+32)+%2Baction_status:(0)+%2Bis_active:(true)+%2Bdue_date:[2019-05-09T19:30:00Z+TO+2019-05-09T19:30:00Z%2B1DAY]+%2B(is_formtype_active:true)+%2B(appType:1)&shards=s1.example.com:8983/solr/actionscomments|s1r1.example.com:8983/solr/actionscomments,s2.example.com:8983/solr/actionscomments|s2r1.example.com:8983/solr/actionscomments&indent=off&shards.tolerant=true&fl=id&start=0&sort=id+desc,id+desc&fq=&rows=1} hits=0 status=0 QTime=51970
> 2019-05-10 09:48:56.746 INFO  (qtp1239731077-128224) [c:actionscomments s:shard1 r:core_node1 x:actionscomments] o.a.s.c.S.Request [actionscomments]  webapp=/solr path=/select params={q=%2Bproject_id:(2121600+2115171+2104206)%2Brecipient_id:(2834330)+%2Bentity_type:(2)+-action_id:(20+32)+%2Baction_status:(0)+%2Bis_active:(true)+%2Bdue_date:[2019-05-10T00:00:00Z+TO+2019-05-10T00:00:00Z%2B1DAY]&shards=s1.example.com:8983/solr/actionscomments|s1r1.example.com:8983/solr/actionscomments,s2.example.com:8983/solr/actionscomments|s2r1.example.com:8983/solr/actionscomments&indent=off&shards.tolerant=true&fl=id&start=0&sort=id+desc,id+desc&fq=&rows=1} hits=98 status=0 QTime=51402
>
>
> My schema fields below :
>
> <field name="id" type="string" indexed="true" stored="true" required="true" multiValued="false"/>
> <field name="project_id" type="tint" indexed="true" stored="true"/>
> <field name="recipient_id" type="tint" indexed="true" stored="true"/>
> <field name="entity_type" type="tint" indexed="true" stored="true"/>
> <field name="action_id" type="tint" indexed="true" stored="true"/>
> <field name="action_status" type="tint" indexed="true" stored="true"/>
> <field name="is_active" type="boolean" indexed="true" stored="true" />
> <field name="is_formtype_active" type="boolean" indexed="true" stored="true" />
> <field name="appType" type="tint" indexed="true" stored="true" />
> <field name="due_date" type="date" indexed="true" stored="true"/>
>
> What could be a problem here? why the query takes too much time at that time?
>
> Sent from Outlook<http://aka.ms/weboutlook>
>
Reply | Threaded
Open this post in threaded view
|

Re: Solr query takes a too much time in Solr 6.1.0

Erick Erickson
That indicates you’re hitting the queryResultCache, which is also supported by your statement about how fast queries are returned after they’re run once. Look at admin UI>>select core>>stats/plugins>>cache>>queryResultCache and you’ll probably see a very hit ratio, approaching 1.

You also have (in another e-mail) 7 zookeepers for a small system. This is severe overkill and gains you nothing. The only times I’ve seen ZooKeepers need more than three is when there are 100s of nodes.

Best,
Erick

> On May 13, 2019, at 7:10 AM, vishal patel <[hidden email]> wrote:
>
> there are many searching and indexing within a millisecond

Reply | Threaded
Open this post in threaded view
|

Re: Solr query takes a too much time in Solr 6.1.0

Erick Erickson
In reply to this post by Bernd Fehling
Oh, and you can freely set docValues=true _and_ have indexed=true on the same field, Solr will use the right structure for the operations it needs. HOWEVER: if you change that definition you _must_ re-index the entire collection.

> On May 13, 2019, at 1:22 AM, Bernd Fehling <[hidden email]> wrote:
>
> Your "sort" parameter has "sort=id+desc,id+desc".
> 1. It doesn't make sense to have a sort on "id" in descending order twice.
> 2. Be aware that the id field has the highest cadinality.
> 3. To speedup sorting have a separate field with docValues=true for sorting.
>   E.g.
> <field name="id" type="string" indexed="true" stored="true" required="true" multiValued="false" />
> <field name="id_sort" type="string" indexed="false" stored="false" docValues="true" useDocValuesAsStored="false" />
> <copyField source="id" dest="id_sort" />
>
> Regards
> Bernd
>
>
> Am 10.05.19 um 15:32 schrieb vishal patel:
>> We have 2 shards and 2 replicas in Live environment. we have multiple collections.
>> Some times some query takes much time(QTime=52552).  There are so many documents indexing and searching within milliseconds.
>> When we executed the same query again using admin panel, it does not take a much time and it completes within 20 milliseconds.
>> My Solr Logs :
>> 2019-05-10 09:48:56.744 INFO  (qtp1239731077-128223) [c:actionscomments s:shard1 r:core_node1 x:actionscomments] o.a.s.c.S.Request [actionscomments]  webapp=/solr path=/select params={q=%2Bproject_id:(2102117)%2Brecipient_id:(4642365)+%2Bentity_type:(1)+-action_id:(20+32)+%2Baction_status:(0)+%2Bis_active:(true)+%2B(is_formtype_active:true)+%2B(appType:1)&shards=s1.example.com:8983/solr/actionscomments|s1r1.example.com:8983/solr/actionscomments,s2.example.com:8983/solr/actionscomments|s2r1.example.com:8983/solr/actionscomments&indent=off&shards.tolerant=true&fl=id&start=0&sort=id+desc,id+desc&fq=&rows=1} hits=198 status=0 QTime=52552
>> 2019-05-10 09:48:56.744 INFO  (qtp1239731077-127998) [c:actionscomments s:shard1 r:core_node1 x:actionscomments] o.a.s.c.S.Request [actionscomments]  webapp=/solr path=/select params={q=%2Bproject_id:(2102117)%2Brecipient_id:(4642365)+%2Bentity_type:(1)+-action_id:(20+32)+%2Baction_status:(0)+%2Bis_active:(true)+%2Bdue_date:[2019-05-09T19:30:00Z+TO+2019-05-09T19:30:00Z%2B1DAY]+%2B(is_formtype_active:true)+%2B(appType:1)&shards=s1.example.com:8983/solr/actionscomments|s1r1.example.com:8983/solr/actionscomments,s2.example.com:8983/solr/actionscomments|s2r1.example.com:8983/solr/actionscomments&indent=off&shards.tolerant=true&fl=id&start=0&sort=id+desc,id+desc&fq=&rows=1} hits=0 status=0 QTime=51970
>> 2019-05-10 09:48:56.746 INFO  (qtp1239731077-128224) [c:actionscomments s:shard1 r:core_node1 x:actionscomments] o.a.s.c.S.Request [actionscomments]  webapp=/solr path=/select params={q=%2Bproject_id:(2121600+2115171+2104206)%2Brecipient_id:(2834330)+%2Bentity_type:(2)+-action_id:(20+32)+%2Baction_status:(0)+%2Bis_active:(true)+%2Bdue_date:[2019-05-10T00:00:00Z+TO+2019-05-10T00:00:00Z%2B1DAY]&shards=s1.example.com:8983/solr/actionscomments|s1r1.example.com:8983/solr/actionscomments,s2.example.com:8983/solr/actionscomments|s2r1.example.com:8983/solr/actionscomments&indent=off&shards.tolerant=true&fl=id&start=0&sort=id+desc,id+desc&fq=&rows=1} hits=98 status=0 QTime=51402
>> My schema fields below :
>> <field name="id" type="string" indexed="true" stored="true" required="true" multiValued="false"/>
>> <field name="project_id" type="tint" indexed="true" stored="true"/>
>> <field name="recipient_id" type="tint" indexed="true" stored="true"/>
>> <field name="entity_type" type="tint" indexed="true" stored="true"/>
>> <field name="action_id" type="tint" indexed="true" stored="true"/>
>> <field name="action_status" type="tint" indexed="true" stored="true"/>
>> <field name="is_active" type="boolean" indexed="true" stored="true" />
>> <field name="is_formtype_active" type="boolean" indexed="true" stored="true" />
>> <field name="appType" type="tint" indexed="true" stored="true" />
>> <field name="due_date" type="date" indexed="true" stored="true"/>
>> What could be a problem here? why the query takes too much time at that time?
>> Sent from Outlook<http://aka.ms/weboutlook>

Reply | Threaded
Open this post in threaded view
|

Re: Solr query takes a too much time in Solr 6.1.0

Shawn Heisey-2
In reply to this post by vishal patel-2
On 5/13/2019 2:51 AM, vishal patel wrote:
>> Executing an identical query again will likely satisfy the query from Solr's caches.  Solr won't need to talk to the actual index, and it will be REALLY fast.  Even a massively complex query, if it is cached, will be fast.
>
> All caches are disabled in our schema file because of our indexing and searching ratio is high in our live environment.

Solr's caches are defined in solrconfig.xml, not the schema.  I mention
this so you can be sure you have the config you think you have.

If your caches are in fact disabled, I am betting that the index data
relevant to that query is getting pushed out of your OS disk cache.
When you execute the same query twice, all the data required by Lucene
to execute that query is available in the OS disk cache, so the second
time is quick because Lucene is pulling the information from RAM, which
is MUCH faster than disk.

Fixing that usually requires adding more memory to the server.

Thanks,
Shawn