RamDirectory and FS at the same moment

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

RamDirectory and FS at the same moment

rafal janik
Hi all!
i've just started my adventure with Lucene i've got one question  
regarding indexing.
Does Lucene have got built-in mechanism to store indexes first in RAM
and after some time or after some number of documents added to move them
to FS? And searching docs all the time in both  'storages' ?
I know the one reason is to use RAMDirectory  and FS and move the
indexes 'by hand' but to do it i need two IndexWriters and two
indexSearchers so  i have to  join results. Is there any better way to
do it?

rafal

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: RamDirectory and FS at the same moment

Anshum-2
Hi Rafal,

If what I understand about your implementation is correct, you could try a
parallelmultisearcher
http://lucene.apache.org/java/2_9_1/api/core/org/apache/lucene/search/ParallelMultiSearcher.html

--
Anshum Gupta
Naukri Labs!
http://ai-cafe.blogspot.com

The facts expressed here belong to everybody, the opinions to me. The
distinction is yours to draw............


2009/11/24 Rafal Janik <[hidden email]>

> Hi all!
> i've just started my adventure with Lucene i've got one question  regarding
> indexing.
> Does Lucene have got built-in mechanism to store indexes first in RAM and
> after some time or after some number of documents added to move them to FS?
> And searching docs all the time in both  'storages' ?
> I know the one reason is to use RAMDirectory  and FS and move the indexes
> 'by hand' but to do it i need two IndexWriters and two indexSearchers so  i
> have to  join results. Is there any better way to do it?
>
> rafal
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

Re: RamDirectory and FS at the same moment

Michael McCandless-2
In reply to this post by rafal janik
Are you sure you even need to do this (manage separate RAMDir)?  Ie
what's driving this (performance)?

IndexWriter already has its own RAM buffer for newly added docs / deletions.

You can use the new near-realtime API (IndexWriter.getReader) in 2.9
to reopen your IndexReader, for searching the full index plus all
changes made with that IndexWriter instance... in my tests this gives
very good performance even at rather high reopen frequency (10X per
second).

Mike

2009/11/24 Rafal Janik <[hidden email]>:

> Hi all!
> i've just started my adventure with Lucene i've got one question  regarding
> indexing.
> Does Lucene have got built-in mechanism to store indexes first in RAM and
> after some time or after some number of documents added to move them to FS?
> And searching docs all the time in both  'storages' ?
> I know the one reason is to use RAMDirectory  and FS and move the indexes
> 'by hand' but to do it i need two IndexWriters and two indexSearchers so  i
> have to  join results. Is there any better way to do it?
>
> rafal
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: RamDirectory and FS at the same moment

rafal janik
Thanks a lot!
of course performance :)
I used 2.41 and now with 2.9.1 it works:)


Michael McCandless pisze:

> Are you sure you even need to do this (manage separate RAMDir)?  Ie
> what's driving this (performance)?
>
> IndexWriter already has its own RAM buffer for newly added docs / deletions.
>
> You can use the new near-realtime API (IndexWriter.getReader) in 2.9
> to reopen your IndexReader, for searching the full index plus all
> changes made with that IndexWriter instance... in my tests this gives
> very good performance even at rather high reopen frequency (10X per
> second).
>
> Mike
>
> 2009/11/24 Rafal Janik <[hidden email]>:
>  
>> Hi all!
>> i've just started my adventure with Lucene i've got one question  regarding
>> indexing.
>> Does Lucene have got built-in mechanism to store indexes first in RAM and
>> after some time or after some number of documents added to move them to FS?
>> And searching docs all the time in both  'storages' ?
>> I know the one reason is to use RAMDirectory  and FS and move the indexes
>> 'by hand' but to do it i need two IndexWriters and two indexSearchers so  i
>> have to  join results. Is there any better way to do it?
>>
>> rafal
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>
>>    
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
>  


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]