"Missing" Docs in Solr

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

"Missing" Docs in Solr

Brian Lininger
Hi All,
I'm seeing some odd behavior that I'm hoping someone might have encountered
before.  We're using Solr 6.6.6 and very infrequently (happened twice in
the past year) we're getting 0 hits returned for a query that I know should
have results.  We've hit this issue once over the past year in 2 separate
collections (both with a single shard), each with several million
documents, where a query will return 0 hits.  I see a similar query run
5-10s later and it will get the expected # of hits (~1M hits) so I know
that we haven't reindexed a million docs between the two queries.  Besides
that I can see that between the 2 queries we only added 150-200 docs with a
single commit so I don't see how that could affect the results in this
manner.

We have a moderate indexing load during the time we see this, we seen much
higher indexing loads without issue but it's also not idle either.  I've
spent a bunch of time trying to reproduce this, tinkering with queries
because I assumed that the problem had to be with the search query and not
with Solr.  Search times for both queries (those with 0 hits and those with
10k+ hits) are taking 30-40ms.

Anyone run into something like this?  Any ideas on something to look for?

Thanks,
Brian Lininger
Reply | Threaded
Open this post in threaded view
|

Re: "Missing" Docs in Solr

Zheng Lin Edwin Yeo
Hi,

Did you encounter any error message during those occasions where you get 0
hits returned?

Regards,
Edwin

On Fri, 16 Aug 2019 at 06:02, Brian Lininger <[hidden email]>
wrote:

> Hi All,
> I'm seeing some odd behavior that I'm hoping someone might have encountered
> before.  We're using Solr 6.6.6 and very infrequently (happened twice in
> the past year) we're getting 0 hits returned for a query that I know should
> have results.  We've hit this issue once over the past year in 2 separate
> collections (both with a single shard), each with several million
> documents, where a query will return 0 hits.  I see a similar query run
> 5-10s later and it will get the expected # of hits (~1M hits) so I know
> that we haven't reindexed a million docs between the two queries.  Besides
> that I can see that between the 2 queries we only added 150-200 docs with a
> single commit so I don't see how that could affect the results in this
> manner.
>
> We have a moderate indexing load during the time we see this, we seen much
> higher indexing loads without issue but it's also not idle either.  I've
> spent a bunch of time trying to reproduce this, tinkering with queries
> because I assumed that the problem had to be with the search query and not
> with Solr.  Search times for both queries (those with 0 hits and those with
> 10k+ hits) are taking 30-40ms.
>
> Anyone run into something like this?  Any ideas on something to look for?
>
> Thanks,
> Brian Lininger
>
Reply | Threaded
Open this post in threaded view
|

Re: "Missing" Docs in Solr

Alexandre Rafalovitch
In reply to this post by Brian Lininger
I would take the server log for those 10 seconds (plus buffer) and really
try to see if something happens in that period.

I am thinking an unexpected commit, index large, alias switch. That may
help you to narrow down the kind of error.

Another option is whether you got empty result or a connection error. I am
thinking firewall that held on but then dropped a connection.

Both of these are unlikely but since you seem to be stuck....

Regards,
    Alex

On Thu, Aug 15, 2019, 6:02 PM Brian Lininger, <[hidden email]>
wrote:

> Hi All,
> I'm seeing some odd behavior that I'm hoping someone might have encountered
> before.  We're using Solr 6.6.6 and very infrequently (happened twice in
> the past year) we're getting 0 hits returned for a query that I know should
> have results.  We've hit this issue once over the past year in 2 separate
> collections (both with a single shard), each with several million
> documents, where a query will return 0 hits.  I see a similar query run
> 5-10s later and it will get the expected # of hits (~1M hits) so I know
> that we haven't reindexed a million docs between the two queries.  Besides
> that I can see that between the 2 queries we only added 150-200 docs with a
> single commit so I don't see how that could affect the results in this
> manner.
>
> We have a moderate indexing load during the time we see this, we seen much
> higher indexing loads without issue but it's also not idle either.  I've
> spent a bunch of time trying to reproduce this, tinkering with queries
> because I assumed that the problem had to be with the search query and not
> with Solr.  Search times for both queries (those with 0 hits and those with
> 10k+ hits) are taking 30-40ms.
>
> Anyone run into something like this?  Any ideas on something to look for?
>
> Thanks,
> Brian Lininger
>
Reply | Threaded
Open this post in threaded view
|

Re: "Missing" Docs in Solr

Brian Lininger
Yeah, I thought of those same problems at first and expected to find
something but no luck.
There are no errors in the solr log for the hour before/after the time that
we saw the issue, the only warnings I see are "PERFORMANCE WARNING:
Overlapping onDeckSearchers=2" but these are for other Collections.... and
as I understand it, this is really just a load issue not a potential
functional issue.

We're getting a valid response sent back from Solr (the search is logged
with 0 hits) & SolrJ, so it doesn't seem to be a network issue.  We're not
using aliases, but that shouldn't be a problem as updates to aliases are
atomic as I understand them.  GC's also are fine during that period.

It's really weird....

On Fri, Aug 16, 2019 at 3:51 AM Alexandre Rafalovitch <[hidden email]>
wrote:

> I would take the server log for those 10 seconds (plus buffer) and really
> try to see if something happens in that period.
>
> I am thinking an unexpected commit, index large, alias switch. That may
> help you to narrow down the kind of error.
>
> Another option is whether you got empty result or a connection error. I am
> thinking firewall that held on but then dropped a connection.
>
> Both of these are unlikely but since you seem to be stuck....
>
> Regards,
>     Alex
>
> On Thu, Aug 15, 2019, 6:02 PM Brian Lininger, <[hidden email]>
> wrote:
>
> > Hi All,
> > I'm seeing some odd behavior that I'm hoping someone might have
> encountered
> > before.  We're using Solr 6.6.6 and very infrequently (happened twice in
> > the past year) we're getting 0 hits returned for a query that I know
> should
> > have results.  We've hit this issue once over the past year in 2 separate
> > collections (both with a single shard), each with several million
> > documents, where a query will return 0 hits.  I see a similar query run
> > 5-10s later and it will get the expected # of hits (~1M hits) so I know
> > that we haven't reindexed a million docs between the two queries.
> Besides
> > that I can see that between the 2 queries we only added 150-200 docs
> with a
> > single commit so I don't see how that could affect the results in this
> > manner.
> >
> > We have a moderate indexing load during the time we see this, we seen
> much
> > higher indexing loads without issue but it's also not idle either.  I've
> > spent a bunch of time trying to reproduce this, tinkering with queries
> > because I assumed that the problem had to be with the search query and
> not
> > with Solr.  Search times for both queries (those with 0 hits and those
> with
> > 10k+ hits) are taking 30-40ms.
> >
> > Anyone run into something like this?  Any ideas on something to look for?
> >
> > Thanks,
> > Brian Lininger
> >
>


--


*Brian Lininger*
Technical Architect, Infrastructure & Search
*Veeva Systems *
[hidden email]
www.veeva.com

*This email and the information it contains are intended for the intended
recipient only, are confidential and may be privileged information exempt
from disclosure by law.*
*If you have received this email in error, please notify us immediately by
reply email and delete this message from your computer.*
*Please do not retain, copy or distribute this email.*
Reply | Threaded
Open this post in threaded view
|

Re: "Missing" Docs in Solr

Alexandre Rafalovitch
is there several 0 results in a row as an anomaly. Or really just one?

You could nearly add SolrJ code to rerun 0-result query with full debug on
if it is a rare enough event.

Regards,
     Alex

On Fri, Aug 16, 2019, 12:05 PM Brian Lininger, <[hidden email]>
wrote:

> Yeah, I thought of those same problems at first and expected to find
> something but no luck.
> There are no errors in the solr log for the hour before/after the time that
> we saw the issue, the only warnings I see are "PERFORMANCE WARNING:
> Overlapping onDeckSearchers=2" but these are for other Collections.... and
> as I understand it, this is really just a load issue not a potential
> functional issue.
>
> We're getting a valid response sent back from Solr (the search is logged
> with 0 hits) & SolrJ, so it doesn't seem to be a network issue.  We're not
> using aliases, but that shouldn't be a problem as updates to aliases are
> atomic as I understand them.  GC's also are fine during that period.
>
> It's really weird....
>
> On Fri, Aug 16, 2019 at 3:51 AM Alexandre Rafalovitch <[hidden email]>
> wrote:
>
> > I would take the server log for those 10 seconds (plus buffer) and really
> > try to see if something happens in that period.
> >
> > I am thinking an unexpected commit, index large, alias switch. That may
> > help you to narrow down the kind of error.
> >
> > Another option is whether you got empty result or a connection error. I
> am
> > thinking firewall that held on but then dropped a connection.
> >
> > Both of these are unlikely but since you seem to be stuck....
> >
> > Regards,
> >     Alex
> >
> > On Thu, Aug 15, 2019, 6:02 PM Brian Lininger, <[hidden email]>
> > wrote:
> >
> > > Hi All,
> > > I'm seeing some odd behavior that I'm hoping someone might have
> > encountered
> > > before.  We're using Solr 6.6.6 and very infrequently (happened twice
> in
> > > the past year) we're getting 0 hits returned for a query that I know
> > should
> > > have results.  We've hit this issue once over the past year in 2
> separate
> > > collections (both with a single shard), each with several million
> > > documents, where a query will return 0 hits.  I see a similar query run
> > > 5-10s later and it will get the expected # of hits (~1M hits) so I know
> > > that we haven't reindexed a million docs between the two queries.
> > Besides
> > > that I can see that between the 2 queries we only added 150-200 docs
> > with a
> > > single commit so I don't see how that could affect the results in this
> > > manner.
> > >
> > > We have a moderate indexing load during the time we see this, we seen
> > much
> > > higher indexing loads without issue but it's also not idle either.
> I've
> > > spent a bunch of time trying to reproduce this, tinkering with queries
> > > because I assumed that the problem had to be with the search query and
> > not
> > > with Solr.  Search times for both queries (those with 0 hits and those
> > with
> > > 10k+ hits) are taking 30-40ms.
> > >
> > > Anyone run into something like this?  Any ideas on something to look
> for?
> > >
> > > Thanks,
> > > Brian Lininger
> > >
> >
>
>
> --
>
>
> *Brian Lininger*
> Technical Architect, Infrastructure & Search
> *Veeva Systems *
> [hidden email]
> www.veeva.com
>
> *This email and the information it contains are intended for the intended
> recipient only, are confidential and may be privileged information exempt
> from disclosure by law.*
> *If you have received this email in error, please notify us immediately by
> reply email and delete this message from your computer.*
> *Please do not retain, copy or distribute this email.*
>
Reply | Threaded
Open this post in threaded view
|

Re: "Missing" Docs in Solr

Brian Lininger
It's just a single query that results in 0 hits, I had the same thought of
just adding code to retry the query when we get 0 hits (assuming that we
expect there to be hits).  That's likely going to be the interim solution
so that we can get more info when this occurs.  It's hard to triage when
it's only happened 2 times in 7+ years of Solr usage ;)
Thanks!

On Fri, Aug 16, 2019 at 9:58 AM Alexandre Rafalovitch <[hidden email]>
wrote:

> is there several 0 results in a row as an anomaly. Or really just one?
>
> You could nearly add SolrJ code to rerun 0-result query with full debug on
> if it is a rare enough event.
>
> Regards,
>      Alex
>
> On Fri, Aug 16, 2019, 12:05 PM Brian Lininger, <[hidden email]>
> wrote:
>
> > Yeah, I thought of those same problems at first and expected to find
> > something but no luck.
> > There are no errors in the solr log for the hour before/after the time
> that
> > we saw the issue, the only warnings I see are "PERFORMANCE WARNING:
> > Overlapping onDeckSearchers=2" but these are for other Collections....
> and
> > as I understand it, this is really just a load issue not a potential
> > functional issue.
> >
> > We're getting a valid response sent back from Solr (the search is logged
> > with 0 hits) & SolrJ, so it doesn't seem to be a network issue.  We're
> not
> > using aliases, but that shouldn't be a problem as updates to aliases are
> > atomic as I understand them.  GC's also are fine during that period.
> >
> > It's really weird....
> >
> > On Fri, Aug 16, 2019 at 3:51 AM Alexandre Rafalovitch <
> [hidden email]>
> > wrote:
> >
> > > I would take the server log for those 10 seconds (plus buffer) and
> really
> > > try to see if something happens in that period.
> > >
> > > I am thinking an unexpected commit, index large, alias switch. That may
> > > help you to narrow down the kind of error.
> > >
> > > Another option is whether you got empty result or a connection error. I
> > am
> > > thinking firewall that held on but then dropped a connection.
> > >
> > > Both of these are unlikely but since you seem to be stuck....
> > >
> > > Regards,
> > >     Alex
> > >
> > > On Thu, Aug 15, 2019, 6:02 PM Brian Lininger, <
> [hidden email]>
> > > wrote:
> > >
> > > > Hi All,
> > > > I'm seeing some odd behavior that I'm hoping someone might have
> > > encountered
> > > > before.  We're using Solr 6.6.6 and very infrequently (happened twice
> > in
> > > > the past year) we're getting 0 hits returned for a query that I know
> > > should
> > > > have results.  We've hit this issue once over the past year in 2
> > separate
> > > > collections (both with a single shard), each with several million
> > > > documents, where a query will return 0 hits.  I see a similar query
> run
> > > > 5-10s later and it will get the expected # of hits (~1M hits) so I
> know
> > > > that we haven't reindexed a million docs between the two queries.
> > > Besides
> > > > that I can see that between the 2 queries we only added 150-200 docs
> > > with a
> > > > single commit so I don't see how that could affect the results in
> this
> > > > manner.
> > > >
> > > > We have a moderate indexing load during the time we see this, we seen
> > > much
> > > > higher indexing loads without issue but it's also not idle either.
> > I've
> > > > spent a bunch of time trying to reproduce this, tinkering with
> queries
> > > > because I assumed that the problem had to be with the search query
> and
> > > not
> > > > with Solr.  Search times for both queries (those with 0 hits and
> those
> > > with
> > > > 10k+ hits) are taking 30-40ms.
> > > >
> > > > Anyone run into something like this?  Any ideas on something to look
> > for?
> > > >
> > > > Thanks,
> > > > Brian Lininger
> > > >
> > >
> >
> >
> > --
> >
> >
> > *Brian Lininger*
> > Technical Architect, Infrastructure & Search
> > *Veeva Systems *
> > [hidden email]
> > www.veeva.com
> >
> > *This email and the information it contains are intended for the intended
> > recipient only, are confidential and may be privileged information exempt
> > from disclosure by law.*
> > *If you have received this email in error, please notify us immediately
> by
> > reply email and delete this message from your computer.*
> > *Please do not retain, copy or distribute this email.*
> >
>


--


*Brian Lininger*
Technical Architect, Infrastructure & Search
*Veeva Systems *
[hidden email]
www.veeva.com

*This email and the information it contains are intended for the intended
recipient only, are confidential and may be privileged information exempt
from disclosure by law.*
*If you have received this email in error, please notify us immediately by
reply email and delete this message from your computer.*
*Please do not retain, copy or distribute this email.*