IndexWriter and IndexReader open at the same time

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

IndexWriter and IndexReader open at the same time

Greg Love
Hello,

I have an application that gets many delete and write resquests at the
same time.  to avoid opening and closing the IndexWriter and
IndexReader everytime one of them need to do a write operation, i keep
them both open and have a shared lock around them whenever i need to
use them for writing.  everything seems to be working in order, but
i'm not sure if this is a safe thing to do.  please let me know.

thank you,
lavafish
Reply | Threaded
Open this post in threaded view
|

Re: IndexWriter and IndexReader open at the same time

Otis Gospodnetic-2
If you have the Lucene book, look at Chapter 2 (page 59 under section
2.9 (Concurrency, thread-safety, and locking issues) in chapter 2
(Indexing)):

  http://www.lucenebook.com/search?query=concurrency+rules

Also, look at Lucene's Bugzilla, where you'll find a contribution that
helps with concurrent IndexReader/IndexWriter usage.

Otis


--- Greg Love <[hidden email]> wrote:

> Hello,
>
> I have an application that gets many delete and write resquests at
> the
> same time.  to avoid opening and closing the IndexWriter and
> IndexReader everytime one of them need to do a write operation, i
> keep
> them both open and have a shared lock around them whenever i need to
> use them for writing.  everything seems to be working in order, but
> i'm not sure if this is a safe thing to do.  please let me know.
>
> thank you,
> lavafish

Reply | Threaded
Open this post in threaded view
|

Re: IndexWriter and IndexReader open at the same time

Greg Love
Thanx for the feedback Otis.  

I did read that section of the book already. But i'm still not clear
on something...  I know that the only way documents can be deleted
from the index is when i close IndexReader, what worries me is that
stuff gets cached in the ram directory when i do inserts.  What I'm
not sure about is if those inserts go into the index at a random time,
or only when i try to insert something new or call close on
IndexWriter.  If the inserted documents can go into the index on the
hard drive at any random time then what i'm doing is not safe.

In the TheServerSide case study of the book, page 375, they say that
they close the IndexWriter and even point out that they did before
openning the IndexReader and deleting. So that kinda makes me wonder
if i'm safe having an IndexReader with deletions and and IndexWriter
with inserts open at the same time (even though my code never does an
index modifying operation at the same time because they share a lock
in the my code).

thank you

On 8/8/05, Otis Gospodnetic <[hidden email]> wrote:

> If you have the Lucene book, look at Chapter 2 (page 59 under section
> 2.9 (Concurrency, thread-safety, and locking issues) in chapter 2
> (Indexing)):
>
>  http://www.lucenebook.com/search?query=concurrency+rules
>
> Also, look at Lucene's Bugzilla, where you'll find a contribution that
> helps with concurrent IndexReader/IndexWriter usage.
>
> Otis
>
>
> --- Greg Love <[hidden email]> wrote:
>
> > Hello,
> >
> > I have an application that gets many delete and write resquests at
> > the
> > same time.  to avoid opening and closing the IndexWriter and
> > IndexReader everytime one of them need to do a write operation, i
> > keep
> > them both open and have a shared lock around them whenever i need to
> > use them for writing.  everything seems to be working in order, but
> > i'm not sure if this is a safe thing to do.  please let me know.
> >
> > thank you,
> > lavafish
>
>
Reply | Threaded
Open this post in threaded view
|

Re: IndexWriter and IndexReader open at the same time

Doug Cutting
Greg Love wrote:
> In the TheServerSide case study of the book, page 375, they say that
> they close the IndexWriter and even point out that they did before
> openning the IndexReader and deleting. So that kinda makes me wonder
> if i'm safe having an IndexReader with deletions and and IndexWriter
> with inserts open at the same time (even though my code never does an
> index modifying operation at the same time because they share a lock
> in the my code).

You should close the IndexReader you are using for deletions before
opening the IndexWriter you use for additions.  For higher throughput,
queue additions and deletions and process them periodically as batches.
  If you're concerned about an addition followed by a deletion of the
same document getting reversed in the queues, then simply check the
addition queue each time you queue a deletion, and remove any matching
additions.

Doug
Reply | Threaded
Open this post in threaded view
|

Re: IndexWriter and IndexReader open at the same time

Greg Love
yeah, that was something i wanted to avoid doing but I guess i have no
choice.  thanx for the help

On 8/9/05, Doug Cutting <[hidden email]> wrote:

> Greg Love wrote:
> > In the TheServerSide case study of the book, page 375, they say that
> > they close the IndexWriter and even point out that they did before
> > openning the IndexReader and deleting. So that kinda makes me wonder
> > if i'm safe having an IndexReader with deletions and and IndexWriter
> > with inserts open at the same time (even though my code never does an
> > index modifying operation at the same time because they share a lock
> > in the my code).
>
> You should close the IndexReader you are using for deletions before
> opening the IndexWriter you use for additions.  For higher throughput,
> queue additions and deletions and process them periodically as batches.
>  If you're concerned about an addition followed by a deletion of the
> same document getting reversed in the queues, then simply check the
> addition queue each time you queue a deletion, and remove any matching
> additions.
>
> Doug
>