How to use lucene for high search performance ?

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

How to use lucene for high search performance ?

王建新

Hi,
    If I use lucene to execute many search requests at one time, the io operation will be the bottleneck of the performance.
    So I use RAMDirectory to avoid io operation.
    But I found RAMDirectory cannot raise the performance much if the index is big( about 1.2G).
    Could anyone give me any advice to raise the performance for concurrent search operation?
    Thanks.
   
roy
Reply | Threaded
Open this post in threaded view
|

Re: How to use lucene for high search performance ?

Michael McCandless-2

Try InstantiatedIndexWriter/Reader (under contrib/instantiated)?

It consumes more RAM than the RAMDirectory approach, but is faster  
performance.

Mike

PS -- this sort of question should go to java-user in the future.

王建新 wrote:

>
> Hi,
>    If I use lucene to execute many search requests at one time, the  
> io operation will be the bottleneck of the performance.
>    So I use RAMDirectory to avoid io operation.
>    But I found RAMDirectory cannot raise the performance much if the  
> index is big( about 1.2G).
>    Could anyone give me any advice to raise the performance for  
> concurrent search operation?
>    Thanks.
>
> roy

Reply | Threaded
Open this post in threaded view
|

Re: How to use lucene for high search performance ?

王建新
Thank you.

If the index files are very big(10G), I cannot load them to ram in one process.

Shoud I use MutilSearcher to load index files with serval processes?
How about its performance?

by the way, I think only .frq and .tis files need to load in ram.
And it can save some ram.

roy

----- Original Message -----
From: "Michael McCandless" <[hidden email]>
To: <[hidden email]>
Sent: Thursday, July 24, 2008 6:09 PM
Subject: Re: How to use lucene for high search performance ?



Try InstantiatedIndexWriter/Reader (under contrib/instantiated)?

It consumes more RAM than the RAMDirectory approach, but is faster  
performance.

Mike

PS -- this sort of question should go to java-user in the future.

王建新 wrote:

>
> Hi,
>    If I use lucene to execute many search requests at one time, the  
> io operation will be the bottleneck of the performance.
>    So I use RAMDirectory to avoid io operation.
>    But I found RAMDirectory cannot raise the performance much if the  
> index is big( about 1.2G).
>    Could anyone give me any advice to raise the performance for  
> concurrent search operation?
>    Thanks.
>
> roy
Reply | Threaded
Open this post in threaded view
|

Re: How to use lucene for high search performance ?

Michael McCandless-2

Let's move this thread to java-user (CC'd).

王建新 wrote:

> Thank you.
>
> If the index files are very big(10G), I cannot load them to ram in  
> one process.

Ahh OK.

> Shoud I use MutilSearcher to load index files with serval processes?
> How about its performance?

MultiSearcher alone doesn't really scale up -- it just lets you  
combine the results of many Searchables.

Maybe you mean ParallelMlultiSearcher?  That class uses a separate  
thread to search each Searchable, so if you are on a multi core/cpu  
machine that should give a net reduction in latency of each search  
(though I don't have any experience here!).

> by the way, I think only .frq and .tis files need to load in ram.
> And it can save some ram.

You mean you don't use any positions information?  Really the OS  
should do the right thing for you -- it should only cache into its IO  
cache those files that you actually use after which searches should be  
fast.

Mike

>
> roy
>
> ----- Original Message -----
> From: "Michael McCandless" <[hidden email]>
> To: <[hidden email]>
> Sent: Thursday, July 24, 2008 6:09 PM
> Subject: Re: How to use lucene for high search performance ?
>
>
>
> Try InstantiatedIndexWriter/Reader (under contrib/instantiated)?
>
> It consumes more RAM than the RAMDirectory approach, but is faster
> performance.
>
> Mike
>
> PS -- this sort of question should go to java-user in the future.
>
> 王建新 wrote:
>
>>
>> Hi,
>>   If I use lucene to execute many search requests at one time, the
>> io operation will be the bottleneck of the performance.
>>   So I use RAMDirectory to avoid io operation.
>>   But I found RAMDirectory cannot raise the performance much if the
>> index is big( about 1.2G).
>>   Could anyone give me any advice to raise the performance for
>> concurrent search operation?
>>   Thanks.
>>
>> roy

Reply | Threaded
Open this post in threaded view
|

Re: How to use lucene for high search performance ?

王建新
Thanks a lot.

I have an idea, Can I use lucene on a 64bits VM?
In the condition, I can load all index files to ram. Then no io operation, I can execute concurrent search in thread pool.

Its performance will be better?



----- Original Message -----
From: "Michael McCandless" <[hidden email]>
To: <[hidden email]>
Cc: <[hidden email]>
Sent: Saturday, July 26, 2008 4:59 AM
Subject: Re: How to use lucene for high search performance ?



Let's move this thread to java-user (CC'd).

王建新 wrote:

> Thank you.
>
> If the index files are very big(10G), I cannot load them to ram in  
> one process.

Ahh OK.

> Shoud I use MutilSearcher to load index files with serval processes?
> How about its performance?

MultiSearcher alone doesn't really scale up -- it just lets you  
combine the results of many Searchables.

Maybe you mean ParallelMlultiSearcher?  That class uses a separate  
thread to search each Searchable, so if you are on a multi core/cpu  
machine that should give a net reduction in latency of each search  
(though I don't have any experience here!).

> by the way, I think only .frq and .tis files need to load in ram.
> And it can save some ram.

You mean you don't use any positions information?  Really the OS  
should do the right thing for you -- it should only cache into its IO  
cache those files that you actually use after which searches should be  
fast.

Mike

>
> roy
>
> ----- Original Message -----
> From: "Michael McCandless" <[hidden email]>
> To: <[hidden email]>
> Sent: Thursday, July 24, 2008 6:09 PM
> Subject: Re: How to use lucene for high search performance ?
>
>
>
> Try InstantiatedIndexWriter/Reader (under contrib/instantiated)?
>
> It consumes more RAM than the RAMDirectory approach, but is faster
> performance.
>
> Mike
>
> PS -- this sort of question should go to java-user in the future.
>
> 王建新 wrote:
>
>>
>> Hi,
>>   If I use lucene to execute many search requests at one time, the
>> io operation will be the bottleneck of the performance.
>>   So I use RAMDirectory to avoid io operation.
>>   But I found RAMDirectory cannot raise the performance much if the
>> index is big( about 1.2G).
>>   Could anyone give me any advice to raise the performance for
>> concurrent search operation?
>>   Thanks.
>>
>> roy
Reply | Threaded
Open this post in threaded view
|

Re: How to use lucene for high search performance ?

Michael McCandless-2

Yes you can, and that should be fast.

Another thing to try is an SSD -- look at the "Lucene performance  
issues" thread on java-user.

Mike

On Jul 27, 2008, at 11:54 PM, 王建新 wrote:

> Thanks a lot.
>
> I have an idea, Can I use lucene on a 64bits VM?
> In the condition, I can load all index files to ram. Then no io  
> operation, I can execute concurrent search in thread pool.
>
> Its performance will be better?
>
>
>
> ----- Original Message -----
> From: "Michael McCandless" <[hidden email]>
> To: <[hidden email]>
> Cc: <[hidden email]>
> Sent: Saturday, July 26, 2008 4:59 AM
> Subject: Re: How to use lucene for high search performance ?
>
>
>
> Let's move this thread to java-user (CC'd).
>
> 王建新 wrote:
>
>> Thank you.
>>
>> If the index files are very big(10G), I cannot load them to ram in
>> one process.
>
> Ahh OK.
>
>> Shoud I use MutilSearcher to load index files with serval processes?
>> How about its performance?
>
> MultiSearcher alone doesn't really scale up -- it just lets you
> combine the results of many Searchables.
>
> Maybe you mean ParallelMlultiSearcher?  That class uses a separate
> thread to search each Searchable, so if you are on a multi core/cpu
> machine that should give a net reduction in latency of each search
> (though I don't have any experience here!).
>
>> by the way, I think only .frq and .tis files need to load in ram.
>> And it can save some ram.
>
> You mean you don't use any positions information?  Really the OS
> should do the right thing for you -- it should only cache into its IO
> cache those files that you actually use after which searches should be
> fast.
>
> Mike
>
>>
>> roy
>>
>> ----- Original Message -----
>> From: "Michael McCandless" <[hidden email]>
>> To: <[hidden email]>
>> Sent: Thursday, July 24, 2008 6:09 PM
>> Subject: Re: How to use lucene for high search performance ?
>>
>>
>>
>> Try InstantiatedIndexWriter/Reader (under contrib/instantiated)?
>>
>> It consumes more RAM than the RAMDirectory approach, but is faster
>> performance.
>>
>> Mike
>>
>> PS -- this sort of question should go to java-user in the future.
>>
>> 王建新 wrote:
>>
>>>
>>> Hi,
>>>  If I use lucene to execute many search requests at one time, the
>>> io operation will be the bottleneck of the performance.
>>>  So I use RAMDirectory to avoid io operation.
>>>  But I found RAMDirectory cannot raise the performance much if the
>>> index is big( about 1.2G).
>>>  Could anyone give me any advice to raise the performance for
>>> concurrent search operation?
>>>  Thanks.
>>>
>>> roy