"timeAllowed" param with "numFound" having a count value but doc list is empty

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

"timeAllowed" param with "numFound" having a count value but doc list is empty

Mark Robinson
Hi,
When in a sample query I used "timeAllowed" as low as 10mS, I got value for
"numFound" as say 2000, but no docs were returned. But when I increased the
value for timeAllowed to be in seconds, never got this scenario.

I have 2 qns:-
1. Why does numFound have a value like say 2000 or even 6000 but no
documents actually returned. During document collection is calculation of
numFound done first and doc collection later?. Is doc list empty because,by
the time doc collection started the timeAllowed cut off took effect?

2. If I give timeAllowed a value say, 10s or above do you think the above
scenario of valid count displayed in numFound, but doc list empty can ever
occur still, as there is more time before cut-off to retrieve at least one
doc ?

Thanks!
Mark
Reply | Threaded
Open this post in threaded view
|

Re: "timeAllowed" param with "numFound" having a count value but doc list is empty

Dominique Bejean
Hi,

1. Yes, your analysis is correct

2. Yes, it can occurs too with very slow query.

Regards

Dominique

Le mar. 15 sept. 2020 à 15:14, Mark Robinson <[hidden email]> a
écrit :

> Hi,
>
> When in a sample query I used "timeAllowed" as low as 10mS, I got value for
>
> "numFound" as say 2000, but no docs were returned. But when I increased the
>
> value for timeAllowed to be in seconds, never got this scenario.
>
>
>
> I have 2 qns:-
>
> 1. Why does numFound have a value like say 2000 or even 6000 but no
>
> documents actually returned. During document collection is calculation of
>
> numFound done first and doc collection later?. Is doc list empty because,by
>
> the time doc collection started the timeAllowed cut off took effect?
>
>
>
> 2. If I give timeAllowed a value say, 10s or above do you think the above
>
> scenario of valid count displayed in numFound, but doc list empty can ever
>
> occur still, as there is more time before cut-off to retrieve at least one
>
> doc ?
>
>
>
> Thanks!
>
> Mark
>
>
Reply | Threaded
Open this post in threaded view
|

Re: "timeAllowed" param with "numFound" having a count value but doc list is empty

Mark Robinson
Thanks Dominique!
So is this parameter generally recommended or not. I wanted to try with a
value of 10s. We are not using it now.
My goal is to prevent a query from running more than 10s on the solr server
and choking it.

What is the general recommendation.

Thanks!
Mark

On Tue, Sep 15, 2020 at 5:38 PM Dominique Bejean <[hidden email]>
wrote:

> Hi,
>
> 1. Yes, your analysis is correct
>
> 2. Yes, it can occurs too with very slow query.
>
> Regards
>
> Dominique
>
> Le mar. 15 sept. 2020 à 15:14, Mark Robinson <[hidden email]> a
> écrit :
>
> > Hi,
> >
> > When in a sample query I used "timeAllowed" as low as 10mS, I got value
> for
> >
> > "numFound" as say 2000, but no docs were returned. But when I increased
> the
> >
> > value for timeAllowed to be in seconds, never got this scenario.
> >
> >
> >
> > I have 2 qns:-
> >
> > 1. Why does numFound have a value like say 2000 or even 6000 but no
> >
> > documents actually returned. During document collection is calculation of
> >
> > numFound done first and doc collection later?. Is doc list empty
> because,by
> >
> > the time doc collection started the timeAllowed cut off took effect?
> >
> >
> >
> > 2. If I give timeAllowed a value say, 10s or above do you think the above
> >
> > scenario of valid count displayed in numFound, but doc list empty can
> ever
> >
> > occur still, as there is more time before cut-off to retrieve at least
> one
> >
> > doc ?
> >
> >
> >
> > Thanks!
> >
> > Mark
> >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: "timeAllowed" param with "numFound" having a count value but doc list is empty

Bram Van Dam
There are a couple of open issues related to the timeAllowed parameter.
For instance it currently doesn't work on conjunction with the
cursorMark parameter [1]. And on Solr 7 it doesn't work at all [2].

But other than that, when users have a lot of query flexibility, it's a
pretty good idea to limit them somehow. You don't want your users to
blow up your servers.

[1] https://issues.apache.org/jira/browse/SOLR-14413

[2] https://issues.apache.org/jira/browse/SOLR-9882

 - Bram

On 16/09/2020 03:04, Mark Robinson wrote:

> Thanks Dominique!
> So is this parameter generally recommended or not. I wanted to try with a
> value of 10s. We are not using it now.
> My goal is to prevent a query from running more than 10s on the solr server
> and choking it.
>
> What is the general recommendation.
>
> Thanks!
> Mark
>
> On Tue, Sep 15, 2020 at 5:38 PM Dominique Bejean <[hidden email]>
> wrote:
>
>> Hi,
>>
>> 1. Yes, your analysis is correct
>>
>> 2. Yes, it can occurs too with very slow query.
>>
>> Regards
>>
>> Dominique
>>
>> Le mar. 15 sept. 2020 à 15:14, Mark Robinson <[hidden email]> a
>> écrit :
>>
>>> Hi,
>>>
>>> When in a sample query I used "timeAllowed" as low as 10mS, I got value
>> for
>>>
>>> "numFound" as say 2000, but no docs were returned. But when I increased
>> the
>>>
>>> value for timeAllowed to be in seconds, never got this scenario.
>>>
>>>
>>>
>>> I have 2 qns:-
>>>
>>> 1. Why does numFound have a value like say 2000 or even 6000 but no
>>>
>>> documents actually returned. During document collection is calculation of
>>>
>>> numFound done first and doc collection later?. Is doc list empty
>> because,by
>>>
>>> the time doc collection started the timeAllowed cut off took effect?
>>>
>>>
>>>
>>> 2. If I give timeAllowed a value say, 10s or above do you think the above
>>>
>>> scenario of valid count displayed in numFound, but doc list empty can
>> ever
>>>
>>> occur still, as there is more time before cut-off to retrieve at least
>> one
>>>
>>> doc ?
>>>
>>>
>>>
>>> Thanks!
>>>
>>> Mark
>>>
>>>
>>
>

Reply | Threaded
Open this post in threaded view
|

Re: "timeAllowed" param with "numFound" having a count value but doc list is empty

Colvin Cowie
In reply to this post by Mark Robinson
Hi Mark,

If queries taking 10 (or however many) seconds isn't acceptable, then
either you need to a) prevent or optimize those queries, b) improve the
performance of your index, c) use timeAllowed and accept that queries
taking that long may fail or provide incomplete results, or d) a
combination of the above.

If you use timeAllowed then you have to accept the possibility that a query
won't complete within the time allowed. Therefore you need to be able to
deal with the possibility of the query failing or of it returning
incomplete results.

In our use of Solr, if a query exceeds timeAllowed we always treat it as a
failure, even if it might have returned partial results, and return a 5xx
response from our own server since we don't want to serve incomplete
results ever. But you could attempt to return whatever results you do
receive, perhaps with a warning message for your client indicating what
happened.


On Wed, 16 Sep 2020 at 02:05, Mark Robinson <[hidden email]> wrote:

> Thanks Dominique!
> So is this parameter generally recommended or not. I wanted to try with a
> value of 10s. We are not using it now.
> My goal is to prevent a query from running more than 10s on the solr server
> and choking it.
>
> What is the general recommendation.
>
> Thanks!
> Mark
>
> On Tue, Sep 15, 2020 at 5:38 PM Dominique Bejean <
> [hidden email]>
> wrote:
>
> > Hi,
> >
> > 1. Yes, your analysis is correct
> >
> > 2. Yes, it can occurs too with very slow query.
> >
> > Regards
> >
> > Dominique
> >
> > Le mar. 15 sept. 2020 à 15:14, Mark Robinson <[hidden email]> a
> > écrit :
> >
> > > Hi,
> > >
> > > When in a sample query I used "timeAllowed" as low as 10mS, I got value
> > for
> > >
> > > "numFound" as say 2000, but no docs were returned. But when I increased
> > the
> > >
> > > value for timeAllowed to be in seconds, never got this scenario.
> > >
> > >
> > >
> > > I have 2 qns:-
> > >
> > > 1. Why does numFound have a value like say 2000 or even 6000 but no
> > >
> > > documents actually returned. During document collection is calculation
> of
> > >
> > > numFound done first and doc collection later?. Is doc list empty
> > because,by
> > >
> > > the time doc collection started the timeAllowed cut off took effect?
> > >
> > >
> > >
> > > 2. If I give timeAllowed a value say, 10s or above do you think the
> above
> > >
> > > scenario of valid count displayed in numFound, but doc list empty can
> > ever
> > >
> > > occur still, as there is more time before cut-off to retrieve at least
> > one
> > >
> > > doc ?
> > >
> > >
> > >
> > > Thanks!
> > >
> > > Mark
> > >
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: "timeAllowed" param with "numFound" having a count value but doc list is empty

Mark Robinson
In reply to this post by Bram Van Dam
Thanks much Bram!

Best,
Mark

On Wed, Sep 16, 2020 at 3:59 AM Bram Van Dam <[hidden email]> wrote:

> There are a couple of open issues related to the timeAllowed parameter.
> For instance it currently doesn't work on conjunction with the
> cursorMark parameter [1]. And on Solr 7 it doesn't work at all [2].
>
> But other than that, when users have a lot of query flexibility, it's a
> pretty good idea to limit them somehow. You don't want your users to
> blow up your servers.
>
> [1] https://issues.apache.org/jira/browse/SOLR-14413
>
> [2] https://issues.apache.org/jira/browse/SOLR-9882
>
>  - Bram
>
> On 16/09/2020 03:04, Mark Robinson wrote:
> > Thanks Dominique!
> > So is this parameter generally recommended or not. I wanted to try with a
> > value of 10s. We are not using it now.
> > My goal is to prevent a query from running more than 10s on the solr
> server
> > and choking it.
> >
> > What is the general recommendation.
> >
> > Thanks!
> > Mark
> >
> > On Tue, Sep 15, 2020 at 5:38 PM Dominique Bejean <
> [hidden email]>
> > wrote:
> >
> >> Hi,
> >>
> >> 1. Yes, your analysis is correct
> >>
> >> 2. Yes, it can occurs too with very slow query.
> >>
> >> Regards
> >>
> >> Dominique
> >>
> >> Le mar. 15 sept. 2020 à 15:14, Mark Robinson <[hidden email]>
> a
> >> écrit :
> >>
> >>> Hi,
> >>>
> >>> When in a sample query I used "timeAllowed" as low as 10mS, I got value
> >> for
> >>>
> >>> "numFound" as say 2000, but no docs were returned. But when I increased
> >> the
> >>>
> >>> value for timeAllowed to be in seconds, never got this scenario.
> >>>
> >>>
> >>>
> >>> I have 2 qns:-
> >>>
> >>> 1. Why does numFound have a value like say 2000 or even 6000 but no
> >>>
> >>> documents actually returned. During document collection is calculation
> of
> >>>
> >>> numFound done first and doc collection later?. Is doc list empty
> >> because,by
> >>>
> >>> the time doc collection started the timeAllowed cut off took effect?
> >>>
> >>>
> >>>
> >>> 2. If I give timeAllowed a value say, 10s or above do you think the
> above
> >>>
> >>> scenario of valid count displayed in numFound, but doc list empty can
> >> ever
> >>>
> >>> occur still, as there is more time before cut-off to retrieve at least
> >> one
> >>>
> >>> doc ?
> >>>
> >>>
> >>>
> >>> Thanks!
> >>>
> >>> Mark
> >>>
> >>>
> >>
> >
>
>
Reply | Threaded
Open this post in threaded view
|

Re: "timeAllowed" param with "numFound" having a count value but doc list is empty

Mark Robinson
In reply to this post by Colvin Cowie
Thanks Colvin!
All the responses were helpful.

Best
Mark

On Wed, Sep 16, 2020 at 4:06 AM Colvin Cowie <[hidden email]>
wrote:

> Hi Mark,
>
> If queries taking 10 (or however many) seconds isn't acceptable, then
> either you need to a) prevent or optimize those queries, b) improve the
> performance of your index, c) use timeAllowed and accept that queries
> taking that long may fail or provide incomplete results, or d) a
> combination of the above.
>
> If you use timeAllowed then you have to accept the possibility that a query
> won't complete within the time allowed. Therefore you need to be able to
> deal with the possibility of the query failing or of it returning
> incomplete results.
>
> In our use of Solr, if a query exceeds timeAllowed we always treat it as a
> failure, even if it might have returned partial results, and return a 5xx
> response from our own server since we don't want to serve incomplete
> results ever. But you could attempt to return whatever results you do
> receive, perhaps with a warning message for your client indicating what
> happened.
>
>
> On Wed, 16 Sep 2020 at 02:05, Mark Robinson <[hidden email]>
> wrote:
>
> > Thanks Dominique!
> > So is this parameter generally recommended or not. I wanted to try with a
> > value of 10s. We are not using it now.
> > My goal is to prevent a query from running more than 10s on the solr
> server
> > and choking it.
> >
> > What is the general recommendation.
> >
> > Thanks!
> > Mark
> >
> > On Tue, Sep 15, 2020 at 5:38 PM Dominique Bejean <
> > [hidden email]>
> > wrote:
> >
> > > Hi,
> > >
> > > 1. Yes, your analysis is correct
> > >
> > > 2. Yes, it can occurs too with very slow query.
> > >
> > > Regards
> > >
> > > Dominique
> > >
> > > Le mar. 15 sept. 2020 à 15:14, Mark Robinson <[hidden email]>
> a
> > > écrit :
> > >
> > > > Hi,
> > > >
> > > > When in a sample query I used "timeAllowed" as low as 10mS, I got
> value
> > > for
> > > >
> > > > "numFound" as say 2000, but no docs were returned. But when I
> increased
> > > the
> > > >
> > > > value for timeAllowed to be in seconds, never got this scenario.
> > > >
> > > >
> > > >
> > > > I have 2 qns:-
> > > >
> > > > 1. Why does numFound have a value like say 2000 or even 6000 but no
> > > >
> > > > documents actually returned. During document collection is
> calculation
> > of
> > > >
> > > > numFound done first and doc collection later?. Is doc list empty
> > > because,by
> > > >
> > > > the time doc collection started the timeAllowed cut off took effect?
> > > >
> > > >
> > > >
> > > > 2. If I give timeAllowed a value say, 10s or above do you think the
> > above
> > > >
> > > > scenario of valid count displayed in numFound, but doc list empty can
> > > ever
> > > >
> > > > occur still, as there is more time before cut-off to retrieve at
> least
> > > one
> > > >
> > > > doc ?
> > > >
> > > >
> > > >
> > > > Thanks!
> > > >
> > > > Mark
> > > >
> > > >
> > >
> >
>