Newbie question: lucene sorting problems and stored fields

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Newbie question: lucene sorting problems and stored fields

alan.boshier
Hi

We are seeing intermittent problems with searches that use
sorted fields (in lucene 1.4.3).

If we add the fields to our Documents as 'unstored' then
we start to see results that have been sorted by Document ID.

The problem goes away if we add the fields as 'stored'.

Is it a requirement when creating a field for sorting to
make it stored?

Thanks

Alan

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: Newbie question: lucene sorting problems and stored fields

Mordo, Aviran (EXP N-NANNATEK)
AFIK, the field has to be indexed, but I don't think it has to be stored
(but then again maybe I'm wrong)

Aviran
http://www.aviransplace.com 

-----Original Message-----
From: Alan Boshier [mailto:[hidden email]]
Sent: Thursday, September 14, 2006 11:39 AM
To: [hidden email]
Subject: Newbie question: lucene sorting problems and stored fields

Hi

We are seeing intermittent problems with searches that use sorted fields
(in lucene 1.4.3).

If we add the fields to our Documents as 'unstored' then we start to see
results that have been sorted by Document ID.

The problem goes away if we add the fields as 'stored'.

Is it a requirement when creating a field for sorting to make it stored?

Thanks

Alan

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: Newbie question: lucene sorting problems and stored fields

alan.boshier
In reply to this post by alan.boshier

That was my understanding (that they had to be indexed) but
making them stored seems to have fixed the problem we were
seeing, which is odd.

Not being an expert on how lucene works internally, I'm
struggling to see how this change could have made any
difference.

-----Original Message-----
From: Mordo, Aviran (EXP N-NANNATEK) [mailto:[hidden email]]
Sent: Thu 9/14/2006 17:18
To: [hidden email]
Cc:
Subject: RE: Newbie question: lucene sorting problems and stored fields

AFIK, the field has to be indexed, but I don't think it has to be stored
(but then again maybe I'm wrong)

Aviran
http://www.aviransplace.com 

-----Original Message-----
From: Alan Boshier [mailto:[hidden email]]
Sent: Thursday, September 14, 2006 11:39 AM
To: [hidden email]
Subject: Newbie question: lucene sorting problems and stored fields

Hi

We are seeing intermittent problems with searches that use sorted fields
(in lucene 1.4.3).

If we add the fields to our Documents as 'unstored' then we start to see
results that have been sorted by Document ID.

The problem goes away if we add the fields as 'stored'.

Is it a requirement when creating a field for sorting to make it stored?

Thanks

Alan

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]






---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Newbie question: lucene sorting problems and stored fields

Yonik Seeley-2
In reply to this post by alan.boshier
On 9/14/06, Alan Boshier <[hidden email]> wrote:
> Is it a requirement when creating a field for sorting to
> make it stored?

No, stored doesn't matter... it must be indexed though.

-Yonik
http://incubator.apache.org/solr Solr, the open-source Lucene search server

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Newbie question: lucene sorting problems and stored fields

Erick Erickson
In reply to this post by alan.boshier
from the ****2.0**** javadoc, the Sort class, so I don't know if it
applies.....

<<<The field must be indexed, but should not be tokenized, and does not need
to be stored (unless you happen to want it back with the rest of your
document data).>>>

Is it possible you're tokenizing it? I'm at a loss as to why *storing* it
would change the behavior, but I guess it's a possibility.

Erick

On 9/14/06, Alan Boshier <[hidden email]> wrote:

>
>
> That was my understanding (that they had to be indexed) but
> making them stored seems to have fixed the problem we were
> seeing, which is odd.
>
> Not being an expert on how lucene works internally, I'm
> struggling to see how this change could have made any
> difference.
>
> -----Original Message-----
> From:   Mordo, Aviran (EXP N-NANNATEK) [mailto:[hidden email]]
> Sent:   Thu 9/14/2006 17:18
> To:     [hidden email]
> Cc:
> Subject:        RE: Newbie question: lucene sorting problems and stored
> fields
>
> AFIK, the field has to be indexed, but I don't think it has to be stored
> (but then again maybe I'm wrong)
>
> Aviran
> http://www.aviransplace.com
>
> -----Original Message-----
> From: Alan Boshier [mailto:[hidden email]]
> Sent: Thursday, September 14, 2006 11:39 AM
> To: [hidden email]
> Subject: Newbie question: lucene sorting problems and stored fields
>
> Hi
>
> We are seeing intermittent problems with searches that use sorted fields
> (in lucene 1.4.3).
>
> If we add the fields to our Documents as 'unstored' then we start to see
> results that have been sorted by Document ID.
>
> The problem goes away if we add the fields as 'stored'.
>
> Is it a requirement when creating a field for sorting to make it stored?
>
> Thanks
>
> Alan
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Newbie question: lucene sorting problems and stored fields

Yonik Seeley-2
In reply to this post by alan.boshier
On 9/14/06, Alan Boshier <[hidden email]> wrote:
> That was my understanding (that they had to be indexed) but
> making them stored seems to have fixed the problem we were
> seeing, which is odd.
>
> Not being an expert on how lucene works internally, I'm
> struggling to see how this change could have made any
> difference.

The most likely explanation is that when you are turning off "stored"
you are really turning off "indexed" by mistake.

-Yonik
http://incubator.apache.org/solr Solr, the open-source Lucene search server

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: Newbie question: lucene sorting problems and stored fields

alan.boshier
In reply to this post by alan.boshier

Thanks for your help - I think I may have stumbled on the
answer but if someone can confirm it I would be most grateful.

My guess is that, if we do the following

1. Retrieve a Document instance D from the index using
   e.g. IndexSearcher.search()
2. Delete the original Document corresponding to D from
   the index using IndexReader.delete()
3. Re-add the Document instance D to the index using
   IndexWriter.addDocument()

then the document we add back will not contain any of the
'unstored' fields that were present in the original document,
because presumably when you retrieve the Document instance
in step 1, the instance returned does not contain fields marked
as 'not stored'.

Hence in our case once the Document went through the steps
above, it lost its sortable fields. Once we marked the sortable
fields as 'stored' then the process above would preserve those
fields and hence sorting would proceed as normal.

Does that make sense?

Thanks

Alan

-----Original Message-----
From: Erick Erickson [mailto:[hidden email]]
Sent: Thu 9/14/2006 18:00
To: [hidden email]
Cc:
Subject: Re: Newbie question: lucene sorting problems and stored fields

from the ****2.0**** javadoc, the Sort class, so I don't know if it
applies.....

<<<The field must be indexed, but should not be tokenized, and does not need
to be stored (unless you happen to want it back with the rest of your
document data).>>>

Is it possible you're tokenizing it? I'm at a loss as to why *storing* it
would change the behavior, but I guess it's a possibility.

Erick

On 9/14/06, Alan Boshier <[hidden email]> wrote:

>
>
> That was my understanding (that they had to be indexed) but
> making them stored seems to have fixed the problem we were
> seeing, which is odd.
>
> Not being an expert on how lucene works internally, I'm
> struggling to see how this change could have made any
> difference.
>
> -----Original Message-----
> From:   Mordo, Aviran (EXP N-NANNATEK) [mailto:[hidden email]]
> Sent:   Thu 9/14/2006 17:18
> To:     [hidden email]
> Cc:
> Subject:        RE: Newbie question: lucene sorting problems and stored
> fields
>
> AFIK, the field has to be indexed, but I don't think it has to be stored
> (but then again maybe I'm wrong)
>
> Aviran
> http://www.aviransplace.com
>
> -----Original Message-----
> From: Alan Boshier [mailto:[hidden email]]
> Sent: Thursday, September 14, 2006 11:39 AM
> To: [hidden email]
> Subject: Newbie question: lucene sorting problems and stored fields
>
> Hi
>
> We are seeing intermittent problems with searches that use sorted fields
> (in lucene 1.4.3).
>
> If we add the fields to our Documents as 'unstored' then we start to see
> results that have been sorted by Document ID.
>
> The problem goes away if we add the fields as 'stored'.
>
> Is it a requirement when creating a field for sorting to make it stored?
>
> Thanks
>
> Alan
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>




---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: Newbie question: lucene sorting problems and stored fields

Doron Cohen
Yes this is correct - updating the doc this way the unstored fields are
getting lost.
This update scenario was also discussed in
http://www.nabble.com/Updating-documents-tf70183.html#a189951.

"Alan Boshier" <[hidden email]> wrote on 14/09/2006 15:39:31:

>
> Thanks for your help - I think I may have stumbled on the
> answer but if someone can confirm it I would be most grateful.
>
> My guess is that, if we do the following
>
> 1. Retrieve a Document instance D from the index using
>    e.g. IndexSearcher.search()
> 2. Delete the original Document corresponding to D from
>    the index using IndexReader.delete()
> 3. Re-add the Document instance D to the index using
>    IndexWriter.addDocument()
>
> then the document we add back will not contain any of the
> 'unstored' fields that were present in the original document,
> because presumably when you retrieve the Document instance
> in step 1, the instance returned does not contain fields marked
> as 'not stored'.
>
> Hence in our case once the Document went through the steps
> above, it lost its sortable fields. Once we marked the sortable
> fields as 'stored' then the process above would preserve those
> fields and hence sorting would proceed as normal.
>
> Does that make sense?
>
> Thanks
>
> Alan
>
> -----Original Message-----
> From:   Erick Erickson [mailto:[hidden email]]
> Sent:   Thu 9/14/2006 18:00
> To:   [hidden email]
> Cc:
> Subject:   Re: Newbie question: lucene sorting problems and stored fields
>
> from the ****2.0**** javadoc, the Sort class, so I don't know if it
> applies.....
>
> <<<The field must be indexed, but should not be tokenized, and does not
need

> to be stored (unless you happen to want it back with the rest of your
> document data).>>>
>
> Is it possible you're tokenizing it? I'm at a loss as to why *storing* it
> would change the behavior, but I guess it's a possibility.
>
> Erick
>
> On 9/14/06, Alan Boshier <[hidden email]> wrote:
> >
> >
> > That was my understanding (that they had to be indexed) but
> > making them stored seems to have fixed the problem we were
> > seeing, which is odd.
> >
> > Not being an expert on how lucene works internally, I'm
> > struggling to see how this change could have made any
> > difference.
> >
> > -----Original Message-----
> > From:   Mordo, Aviran (EXP N-NANNATEK) [mailto:[hidden email]]
> > Sent:   Thu 9/14/2006 17:18
> > To:     [hidden email]
> > Cc:
> > Subject:        RE: Newbie question: lucene sorting problems and stored
> > fields
> >
> > AFIK, the field has to be indexed, but I don't think it has to be
stored

> > (but then again maybe I'm wrong)
> >
> > Aviran
> > http://www.aviransplace.com
> >
> > -----Original Message-----
> > From: Alan Boshier [mailto:[hidden email]]
> > Sent: Thursday, September 14, 2006 11:39 AM
> > To: [hidden email]
> > Subject: Newbie question: lucene sorting problems and stored fields
> >
> > Hi
> >
> > We are seeing intermittent problems with searches that use sorted
fields
> > (in lucene 1.4.3).
> >
> > If we add the fields to our Documents as 'unstored' then we start to
see
> > results that have been sorted by Document ID.
> >
> > The problem goes away if we add the fields as 'stored'.
> >
> > Is it a requirement when creating a field for sorting to make it
stored?

> >
> > Thanks
> >
> > Alan
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]
> >
> >
> >
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]
> >
> >
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Loading...