Sorting/paging problem

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Sorting/paging problem

Charlie Jackson
I've run into a strange issue with my Solr installation. I'm running
queries that are sorting by a DateField field but from time to time, I'm
seeing individual records very much out of order. What's more, they
appear on multiple pages of my result set. Let me give an example.
Starting with a basic query, I sort on the date that the document was
added to the index and see these rows on the first page (I'm just
showing the date field here):

 

<doc><date name="indexed_date">2009-09-23T19:24:47.419Z</date></doc>

<doc><date name="indexed_date">2009-09-23T19:25:03.229Z</date></doc>

<doc><date name="indexed_date">2009-09-23T19:25:03.400Z</date></doc>

<doc><date name="indexed_date">2009-09-23T19:25:19.951</date></doc>

<doc><date name="indexed_date">2009-09-23T20:10:07.919Z</date></doc>

 

Note how the last document's date jumps a bit. Not necessarily a
problem, but the next page looks this:

 

<doc><date name="indexed_date">2009-09-23T19:26:16.022Z</date></doc>

<doc><date name="indexed_date">2009-09-23T19:26:32.547Z</date></doc>

<doc><date name="indexed_date">2009-09-23T19:27:45.470Z</date></doc>

<doc><date name="indexed_date">2009-09-23T19:27:45.592Z</date></doc>

<doc><date name="indexed_date">2009-09-23T20:10:07.919Z</date></doc>

 

So, not only is the date sorting wrong, but the exact same document
shows up on the next page, also still out of date order. I've seen the
same document show up in 4-5 pages in some cases. It's always the last
record on the page, too. If I change the page size, the problem seems to
disappear for a while, but then starts up again later. Also, running the
same query/queries later on doesn't show the same behavior.

 

Could it be some sort of page boundary issue with the cache? Has anyone
else run into a problem like this? I'm using the Sept 22 nightly build.

 

- Charlie

Reply | Threaded
Open this post in threaded view
|

Re: Sorting/paging problem

Lance Norskog-2
Which version of Java are you using?

Please try the standard tricks:
Do a fresh checkout of the Solr trunk.
Do 'ant clean dist' and use the newly built war & latest lucene libraries.
Try changing the JVM startup parameters which control how incremental
compilation works: -server and others. Also try changing the garbage
collection algorithms.

On Thu, Sep 24, 2009 at 9:49 AM, Charlie Jackson
<[hidden email]> wrote:

> I've run into a strange issue with my Solr installation. I'm running
> queries that are sorting by a DateField field but from time to time, I'm
> seeing individual records very much out of order. What's more, they
> appear on multiple pages of my result set. Let me give an example.
> Starting with a basic query, I sort on the date that the document was
> added to the index and see these rows on the first page (I'm just
> showing the date field here):
>
>
>
> <doc><date name="indexed_date">2009-09-23T19:24:47.419Z</date></doc>
>
> <doc><date name="indexed_date">2009-09-23T19:25:03.229Z</date></doc>
>
> <doc><date name="indexed_date">2009-09-23T19:25:03.400Z</date></doc>
>
> <doc><date name="indexed_date">2009-09-23T19:25:19.951</date></doc>
>
> <doc><date name="indexed_date">2009-09-23T20:10:07.919Z</date></doc>
>
>
>
> Note how the last document's date jumps a bit. Not necessarily a
> problem, but the next page looks this:
>
>
>
> <doc><date name="indexed_date">2009-09-23T19:26:16.022Z</date></doc>
>
> <doc><date name="indexed_date">2009-09-23T19:26:32.547Z</date></doc>
>
> <doc><date name="indexed_date">2009-09-23T19:27:45.470Z</date></doc>
>
> <doc><date name="indexed_date">2009-09-23T19:27:45.592Z</date></doc>
>
> <doc><date name="indexed_date">2009-09-23T20:10:07.919Z</date></doc>
>
>
>
> So, not only is the date sorting wrong, but the exact same document
> shows up on the next page, also still out of date order. I've seen the
> same document show up in 4-5 pages in some cases. It's always the last
> record on the page, too. If I change the page size, the problem seems to
> disappear for a while, but then starts up again later. Also, running the
> same query/queries later on doesn't show the same behavior.
>
>
>
> Could it be some sort of page boundary issue with the cache? Has anyone
> else run into a problem like this? I'm using the Sept 22 nightly build.
>
>
>
> - Charlie
>
>



--
Lance Norskog
[hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Sorting/paging problem

hossman
In reply to this post by Charlie Jackson

: <doc><date name="indexed_date">2009-09-23T19:25:03.400Z</date></doc>
:
: <doc><date name="indexed_date">2009-09-23T19:25:19.951</date></doc>
:
: <doc><date name="indexed_date">2009-09-23T20:10:07.919Z</date></doc>

is that a cut/paste error, or did you really get a date back from Solr
w/o the trailing "Z" ?!?!?!

        ...

: So, not only is the date sorting wrong, but the exact same document
: shows up on the next page, also still out of date order. I've seen the
: same document show up in 4-5 pages in some cases. It's always the last
: record on the page, too. If I change the page size, the problem seems to

that is really freaking weird.  can you reproduce this in a simple
example?  maybe an index that's small enough (and doesn't contain
confidential information) that you could zip up and post online?



-Hoss

Reply | Threaded
Open this post in threaded view
|

RE: Sorting/paging problem

Charlie Jackson
Oops, the missing trailing Z was probably just a cut and paste error.

It might be tough to come up with a case that can reproduce it -- it's a
sticky issue. I'll post it if I can, though.


-----Original Message-----
From: Chris Hostetter [mailto:[hidden email]]
Sent: Tuesday, September 29, 2009 6:08 PM
To: [hidden email]
Subject: Re: Sorting/paging problem


: <doc><date name="indexed_date">2009-09-23T19:25:03.400Z</date></doc>
:
: <doc><date name="indexed_date">2009-09-23T19:25:19.951</date></doc>
:
: <doc><date name="indexed_date">2009-09-23T20:10:07.919Z</date></doc>

is that a cut/paste error, or did you really get a date back from Solr
w/o the trailing "Z" ?!?!?!

        ...

: So, not only is the date sorting wrong, but the exact same document
: shows up on the next page, also still out of date order. I've seen the
: same document show up in 4-5 pages in some cases. It's always the last
: record on the page, too. If I change the page size, the problem seems
to

that is really freaking weird.  can you reproduce this in a simple
example?  maybe an index that's small enough (and doesn't contain
confidential information) that you could zip up and post online?



-Hoss