how to safely periodically reopen the IndexReader?

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

how to safely periodically reopen the IndexReader?

zaexage
Hi, all

I've got a question here needing your help:
For my index, I opened one IndexWriter (autocommit mode) and one
IndexSearcher on it;
I have quite a lot of threads here concurrently writing new documents and
doing search in index.
the IndexWriter is flushed periodically.
as I want the IndexSearcher be able to see the latest record, I also need to
periodically reopen the underlying IndexReader.

But is it safe to just call close() on the IndexSearcher when there are
still threads using it?

or maybe I should use atomic counter to make sure there's no thread using it
before calling close() on IndexSearcher?

thanks.
Reply | Threaded
Open this post in threaded view
|

RE: how to safely periodically reopen the IndexReader?

John Griffin-3
Your users won't appreciate your closing the searcher on them. That is, if
you have a highly concurrent system.

I don't know about 2.3.0 yet. Haven't had much chance to see the changes but
with 2.2.0 I use an atomic counter. It's not that much to program.

Regards,
John G.


-----Original Message-----
From: ZaeX [mailto:[hidden email]]
Sent: Saturday, February 16, 2008 2:32 AM
To: [hidden email]
Subject: how to safely periodically reopen the IndexReader?

Hi, all

I've got a question here needing your help:
For my index, I opened one IndexWriter (autocommit mode) and one
IndexSearcher on it;
I have quite a lot of threads here concurrently writing new documents and
doing search in index.
the IndexWriter is flushed periodically.
as I want the IndexSearcher be able to see the latest record, I also need to
periodically reopen the underlying IndexReader.

But is it safe to just call close() on the IndexSearcher when there are
still threads using it?

or maybe I should use atomic counter to make sure there's no thread using it
before calling close() on IndexSearcher?

thanks.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: how to safely periodically reopen the IndexReader?

Robert.Hastings
In reply to this post by zaexage
We have the same situation and use an atomic counter.  Basically, we have
a SearcherHolder class and a SearcherManager class.  The SearcherHolder
holds the searcher and the number of threads referencing the searcher.

When the thread that writes to the index closes the index, it sends an
event that the SearcherManager is listening for.  The next time a search
is performed the manager opens a new searcher.  The old searcher is marked
as outdated but is not closed until the last thread that references the
searcher returns it.





ZaeX <[hidden email]>
02/16/2008 03:32 AM
Please respond to
[hidden email]


To
[hidden email]
cc

Subject
how to safely periodically reopen the IndexReader?






Hi, all

I've got a question here needing your help:
For my index, I opened one IndexWriter (autocommit mode) and one
IndexSearcher on it;
I have quite a lot of threads here concurrently writing new documents and
doing search in index.
the IndexWriter is flushed periodically.
as I want the IndexSearcher be able to see the latest record, I also need
to
periodically reopen the underlying IndexReader.

But is it safe to just call close() on the IndexSearcher when there are
still threads using it?

or maybe I should use atomic counter to make sure there's no thread using
it
before calling close() on IndexSearcher?

thanks.

Reply | Threaded
Open this post in threaded view
|

Re: how to safely periodically reopen the IndexReader?

zaexage
Yes, I'm using a solution quite similar to this one. :)

But when I read the Lucene 2.3 IndexReader.java code and javadoc, I
see they have added a synchronous reference-counter to IndexReader and
when it drops to zero, this instance will automatically call doClose()
to clean, but the manipulator methods of this counter is marked as
`protected'.
I thought of extending FilterIndexReader to use the inherited counter
manipulator. Has anyone tried this way?

2008/2/19, [hidden email] <[hidden email]>:

> We have the same situation and use an atomic counter. Basically, we have
> a SearcherHolder class and a SearcherManager class. The SearcherHolder
> holds the searcher and the number of threads referencing the searcher.
>
> When the thread that writes to the index closes the index, it sends an
> event that the SearcherManager is listening for. The next time a search
> is performed the manager opens a new searcher. The old searcher is marked
> as outdated but is not closed until the last thread that references the
> searcher returns it.
>
>
>
>
>
> ZaeX <[hidden email]>
> 02/16/2008 03:32 AM
> Please respond to
> [hidden email]
>
>
> To
> [hidden email]
> cc
>
> Subject
> how to safely periodically reopen the IndexReader?
>
>
>
>
>
>
> Hi, all
>
> I've got a question here needing your help:
> For my index, I opened one IndexWriter (autocommit mode) and one
> IndexSearcher on it;
> I have quite a lot of threads here concurrently writing new documents and
> doing search in index.
> the IndexWriter is flushed periodically.
> as I want the IndexSearcher be able to see the latest record, I also need
> to
> periodically reopen the underlying IndexReader.
>
> But is it safe to just call close() on the IndexSearcher when there are
> still threads using it?
>
> or maybe I should use atomic counter to make sure there's no thread using
> it
> before calling close() on IndexSearcher?
>
> thanks.
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: how to safely periodically reopen the IndexReader?

Stephane Nicoll
In reply to this post by Robert.Hastings
On Mon, Feb 18, 2008 at 6:08 PM,  <[hidden email]> wrote:
> We have the same situation and use an atomic counter.  Basically, we have
>  a SearcherHolder class and a SearcherManager class.  The SearcherHolder
>  holds the searcher and the number of threads referencing the searcher.
>
>  When the thread that writes to the index closes the index, it sends an
>  event that the SearcherManager is listening for.  The next time a search
>  is performed the manager opens a new searcher.  The old searcher is marked
>  as outdated but is not closed until the last thread that references the
>  searcher returns it.

That sounds interesting. Couldn't it be part of the core capabilities
of Lucene?

St├ęphane


--
Large Systems Suck: This rule is 100% transitive. If you build one,
you suck" -- S.Yegge

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]